MinHash
A probabilistic algorithm used to quickly estimate the Jaccard similarity between two large sets, heavily used for massive dataset deduplication.
Think of It Like This
Like comparing two massive books by just checking if the first random words on ten specific pages are identical.
MinHash works by applying multiple random hash functions to the sets and keeping the minimum hash value for each. The probability that two sets have the same minimum hash value is exactly equal to their Jaccard similarity. It scales efficiently and is a standard pipeline step when curating terabytes of text for LLM pre-training.