Data Concepts

Data Quality Filtering

At web scale most text is not worth training on. Filtering is what turns a scrape into a corpus, and the filter you choose becomes part of the model itself.

Four filter stages that each keep a plausible-sounding fraction compound to a retention rate of under two percent, so a hundred terabytes of scrape becomes under two terabytes of corpus

Why Does This Exist?

A raw web scrape is mostly not language. It's navigation menus, cookie banners, SEO spam, auto-translated product listings, template boilerplate repeated across a million domains, and long runs of markup.

Train on it directly and you get a model that has learned the shape of a cookie banner. The finding that shaped current practice is blunt: a smaller, filtered corpus beats a larger unfiltered one, at equal compute. Quality per token is the thing being optimised, and filtering is how you buy it.

Which puts a lot of weight on choices nobody writes down. The filter becomes part of the model — every threshold decides what the model will and won't have seen, and a filter that quietly removes a dialect, a domain, or a register removes the model's ability to handle it.

Think of It Like This

Panning for gold, and the size of your sieve

You're panning a river. Most of what comes up is gravel, so you shake the pan and let the light material wash out. What stays is denser and more likely to be worth keeping.

Nobody inspects each grain. The method is statistical: a property that correlates with value — density — does the sorting, and you accept that some gold washes away and some pyrite stays.

Now the part that decides whether you get rich. Your sieve's mesh determines what you can ever find. Too coarse and you keep gravel. Too fine and you keep sand along with the gold, and you've spent the day processing sand. And if the gold in this particular river happens to be unusually fine-grained, a standard sieve throws away exactly the thing you came for, and the pan will never tell you.

How It Actually Works

Filtering is a cascade: cheap filters first on everything, expensive ones last on what survives. Order is an economic decision as much as a quality one.

Stage 1 — cheap and structural

Language identification, which is usually the largest single cut. Length bounds, since a 12-word page and a 500,000-word dump are both usually junk. Character-set and encoding checks. Symbol-to-word ratio, which catches markup and code dumps in a text corpus. Repetition checks, catching pages that repeat a line two hundred times.

Boilerplate and template removal is the highest-value item here and the most often skipped. If a third of every scraped page is the same navigation block, that's a third of the corpus spent teaching the model a menu.

Stage 2 — deduplication

Exact and near-duplicate removal, which is a big enough subject to have its own page. It belongs in the cascade rather than after it, because deduplicating before the expensive classifier means the classifier runs on far fewer documents.

Stage 3 — model-based quality

Perplexity filtering. Score each document with a small language model trained on text you trust, and drop the documents it finds most surprising. The intuition is that high perplexity means "unlike good text". The catch is that it also means "unusual", so this filter is biased toward the register of whatever reference corpus you used, and it will happily remove technical writing, poetry and non-standard dialects.

Classifier-based filtering. Train a fast classifier — often just logistic regression over n-grams — to distinguish a small set of known-good documents from random scrape, and keep the documents it scores highly. Cheap to run at scale, and it's exactly a labelling function: fast, noisy, and encoding whatever your positive set happened to contain.

Toxicity, safety and PII filters. Necessary, and blunt. A toxicity classifier trained on one distribution over-flags reclaimed language and discussion about harm, so an aggressive setting removes the model's ability to talk about the topic at all rather than its willingness to produce it.

Stage 4 — decontamination

Removing anything overlapping your evaluation sets, which is its own concern and runs last because it needs the final corpus.

The measurement that keeps you honest

Log retention per stage, per source and per language, not just overall. An aggregate retention of 30% can hide a language retained at 2% or a domain removed entirely, and the aggregate is the number people report.

The rule worth adopting: any slice retained at a rate very different from the mean is a finding that needs a reason. Sometimes the reason is good. Often it's a threshold calibrated on English prose meeting something that isn't.

Worked example

Start with 100TB of raw scrape and four stages, each keeping a fraction that sounds mild on its own.

Language identification keeps 30%, so 30TB. Heuristics keep 50%, so 15TB. Deduplication keeps 40%, so 6TB. The quality classifier keeps 30%, so 1.8TB.

Overall retention is $0.30 \times 0.50 \times 0.40 \times 0.30 = 0.018$ — 1.8%, a factor of 55.6. Nobody chose to throw away 98% of the data; four reasonable-sounding stages did it multiplicatively.

Two things follow. Small threshold changes compound, so tightening each stage by ten percentage points takes retention to 0.9% and halves your corpus. And because the stages multiply, the cheapest way to get more data is usually loosening the most aggressive stage rather than tuning all four.

Show Me the Code

1import numpy as np2
3raw_tb: float = 100.04stages: dict[str, float] = {5    "language id": 0.30,6    "heuristics": 0.50,7    "deduplication": 0.40,8    "quality classifier": 0.30,9}10
11def cascade(size: float, keeps: dict[str, float]) -> list[float]:12    """Retention compounds, so a mild-looking stage list ends up brutal."""13    out, current = [], size14    for keep in keeps.values():15        current *= keep16        out.append(round(current, 2))17    return out18
19print(cascade(raw_tb, stages))                          # -> [30.0, 15.0, 6.0, 1.8]20print(round(float(np.prod(list(stages.values()))), 5))  # -> 0.018   overall retention21print(round(raw_tb / cascade(raw_tb, stages)[-1], 1))   # -> 55.6    the factor thrown away

Four numbers between 0.3 and 0.5, multiplied, and 98.2% of the data is gone. Worth printing on the wall of any corpus project.

Watch Out For

A filter calibrated on one register, applied to everything

Your perplexity filter's reference model was trained on news and Wikipedia. Its thresholds look sensible on a sample, which was also mostly news and Wikipedia.

Then it meets the rest of the corpus. Legal text has unusual sentence structure, so it scores as high perplexity. So does poetry, so does mathematical writing, so does source code, and so does any non-standard dialect or a language with less representation in the reference model. All of it looks "surprising", all of it gets cut, and the aggregate retention number stays exactly where you expected.

What you've built is a model that can't write a contract, read a proof, or handle a dialect its users speak — and no evaluation you run will say so, because your benchmarks are in the same register as the reference corpus.

Log retention per source, per language and per document type, and treat any slice far from the mean as a finding. Then check the removals by hand: pull 50 rejected documents from a low-retention slice and read them. If they look fine, your threshold is measuring register rather than quality.

Tuning the cascade without holding the compute budget fixed

You loosen the quality classifier, the corpus doubles, the model trains on twice the tokens and scores better. Filtering was too aggressive, apparently.

Except you also doubled the tokens seen, so you changed two things. The whole claim of quality filtering is quality per token, which can only be tested at a fixed token budget — same number of training tokens, different corpora, compare. Do it that way and aggressive filtering usually wins, which is the opposite of what the loose comparison suggested.

The reverse error is just as common: tightening the filter, training on far fewer tokens, scoring worse, and concluding the filter was bad when the model was simply undertrained.

Fix the token budget, vary the filter, and compare. And run the comparison on evaluations that cover the slices your filter affects, because a filter that removes code will look harmless on a benchmark with no code in it.

The Quick Version

Raw web scrape is mostly boilerplate, spam and markup. A smaller filtered corpus beats a larger unfiltered one at equal compute.
The filter becomes part of the model. What it removes, the model can never do.
Run a cascade: cheap structural filters first, deduplication next, model-based quality after, decontamination last.
Boilerplate removal is the highest-value cheap stage and the most often skipped.
Perplexity filtering removes "unusual" as well as "bad", so it's biased toward its reference corpus's register.
Classifier filtering is a labelling function: fast, noisy, and shaped by whatever your positive set contained.
Retention compounds. Four stages at 0.30, 0.50, 0.40 and 0.30 keep 1.8% — a factor of 55.6.
Log retention per source, per language and per document type. An aggregate hides a slice cut to nothing.
Compare filters at a fixed token budget, or you're measuring corpus size instead of quality.