Benchmark Contamination
Benchmark questions live on the public internet, and so does your training scrape. When the two overlap, the score measures memory instead of reasoning.
Why Does This Exist?
Classic evaluation rests on one promise: the test set never informed any decision. You split your data, you lock the test split away, and train, validation and test splits hold. That promise is easy to keep when you own the whole pipeline.
Pretraining broke the promise by changing what "the data" means. A web scrape eats the same internet where benchmarks are published: the questions, the reference answers, the worked solutions in blog posts. Overlap stops being a mistake you might make and becomes the default you must disprove. A model can score 0.595 on a benchmark where its real ability is 0.55, and nothing in that single number says which part was reasoning and which part was recall. That blend is benchmark contamination.
This page covers the judgment: how the leak happens, how you detect it, and what restores trust. The general bug class, which bites small supervised projects too, is data leakage. The pipeline procedure that scrubs a corpus is dataset decontamination. Read this page for whether to believe a score; read that one for how to clean the data behind it.
Think of It Like This
The answer key on the photocopier
Your teacher leaves the answer key in the photocopier, and you study it the night before the exam instead of the textbook. Next morning you score near perfect. The grade says genius; the truth says you read the questions early.
Every part maps. The textbook is the training corpus. The photocopier is the public internet, where benchmarks sit in the open. Studying the key is pretraining: the model does not choose to cheat, it just reads everything. And the inflated grade is your benchmark score.
Where it stops: you knew you cheated, and you saw the exact paper. A model builder usually has no idea the overlap happened, and near-copies count too. A paraphrased question you would still recognize teaches the model just as well.
How It Actually Works
Contamination is a mixture. Call the benchmark size , the number of leaked questions , and the leaked fraction . In plain words: is the share of the test the model already saw during training. Call true ability on unseen questions (accuracy with no prior exposure) and accuracy on leaked questions (accuracy from memory, usually near 1). The score you publish is:
is the reported score. Read the formula as a weighted average: the leaked slice is answered from memory, the rest from ability, and one number hides the blend.
Where the overlap comes from
Benchmarks are public by design, so anyone can run them. That same openness puts the questions and answers into repos, papers, forum threads, and quoted model outputs, all of which end up in scrapes. The leak does not need the official test file either. A paraphrase, a translation, or a blog post working through the solution teaches the same content, and none of them match the original word for word.
How you spot it
Three checks, in order of cost. First, an overlap audit: chop the benchmark into short word sequences and search the corpus for exact matches, a standard choice being 13-word sequences. Any training document sharing one gets flagged. The full procedure lives in dataset decontamination.
Second, verbatim probing: feed the model the first half of a benchmark question and see whether it completes the second half word for word. Reasoning does not reproduce punctuation; memory does. Third, the surprise test: a model that trained on a passage finds it less surprising than fresh text, so benchmark passages with unusually low surprise are suspects. No single check is proof. Together they converge.
Worked example
A benchmark with questions, of which leaked into training, so . On the 180 clean questions the model gets 99 right, so . On the 20 leaked ones it gets all 20 from memory, so .
Plug in:
Or in counts: . True ability is 0.55, the published score is 0.595, and the gap of is memory wearing ability's clothes. Ten percent leaked, four and a half points invented. The diagram above draws exactly this split.
What it costs to fix
Scrubbing is a trade, not a cure. Exact-match filtering is cheap and catches verbatim copies, but every phrase you flag deletes real training data, including harmless text that happens to share a few words. Push it hard and you shrink and skew the corpus to protect one number.
The method also assumes leaks are verbatim, which paraphrases and translations break without trying. So skip the ocean-scrub when your evaluation never touches public data: a private test set written after the training cutoff sidesteps the problem, and when a benchmark postdates the cutoff with clean provenance, overlap is near impossible and auditing buys little.
Code
The point of this snippet is the split: recompute what the score would be without the leaked slice, so one reported number never gets the last word.
def score_split(n_total: int, n_leaked: int, clean_correct: int, leaked_correct: int) -> tuple[float, float]: """Split a benchmark score into true ability and reported score.""" n_clean = n_total - n_leaked true_ability = clean_correct / n_clean reported = (clean_correct + leaked_correct) / n_total return true_ability, reported
true_ability, reported = score_split( n_total=200, n_leaked=20, clean_correct=99, leaked_correct=20)print(f"{true_ability:.3f}") # -> 0.550print(f"{reported:.3f}") # -> 0.595print(f"{reported - true_ability:+.3f}") # -> +0.045Twenty memorized answers buy four and a half points. Any benchmark whose leaked slice you can estimate deserves this arithmetic before you quote its score.
Watch Out For
Paraphrases pass the filter and still inflate the score
Your overlap audit reports zero matches, so you publish with confidence, and the score is still part memory. The cause is usually a reworded leak: a translation of the benchmark, a forum rewrite, a solution walkthrough. Exact sequences never match, but the model learned the content anyway.
Symptom: verbatim probing fails while the model answers benchmark items in suspiciously textbook phrasing, or scores far above its level on similar fresh items. Fix: add a semantic near-duplicate sweep over the corpus, plant canary strings (unique nonsense phrases) in future benchmarks to test for ingestion, and keep one private held-out set that no filter result can touch. If the number matters, that private set is the number.
Scrubbing so hard you starve the training set
The other failure points the opposite way. Determined to show zero overlap, you shorten the match length and delete every document sharing a phrase with the benchmark. Common idioms trip short matches constantly, so large harmless fractions of the corpus go in the bin.
Symptom: the filter's removal log keeps growing while quality on unrelated tasks sags, because you deleted good data to protect one metric. Fix: report the removal rate alongside the score, tune the match length instead of minimizing it, allowlist boilerplate, and delete the specific overlapping documents rather than whole domains. Decontamination protects the measurement; the training set still has a job.
The Quick Version
- Benchmark contamination means test questions leaked into training data, so the score mixes memory with reasoning.
- At web scale overlap is the default, because benchmarks are published on the same internet that pretraining scrapes.
- A 200-question test with 20 leaked items reports 0.595 for a model whose true ability is 0.55.
- Exact-match filtering catches verbatim copies but misses paraphrases, translations, and worked solutions.
- Private held-out sets are the only fix that does not depend on catching every leak; everything else is an estimate.