Efficient Large-Scale Pretraining

Pretraining eats GPU-years most labs never see. Efficient methods — masking, filtering, staging — buy the same eyes for a fraction of the compute.

Filtering to 30 percent of images and 25 percent of tokens cuts attention work more than thirteenfold

Why Does This Exist?

Frontier pretraining bills read like phone numbers: thousands of GPU-years per run, affordable to a handful of labs, repeatable by none of the rest. Research Stratifies into those who train eyes and those who rent them — and rented eyes cannot be opened, fixed, or fully trusted. Carbon and iteration speed argue alongside equity.

Efficient pretraining buys eyes cheaply: mask most tokens (encode the visible few), filter dead-weight images before training, stage resolutions from coarse to fine. Each multiplies the others, and careful combinations reach competitive backbones on academic budgets. The quality-over-quantity ally is data-centric AI; the consumer is video foundation models.

The one idea you need first. Masked image modeling hides $\sim75\%$ of patches and reconstructs them from the visible quarter: the encoder runs on $49$ of $196$ tokens, attention work falls roughly sixteenfold, and the puzzle stays hard because images are redundant. Data filtering keeps the top fraction by learned quality scores (aesthetic, text-alignment, dedup), discarding watermarks and duplicates before they bill GPU-hours.

Think of It Like This

Skimming solved problems, not copying textbooks

Two students face the same textbook. One copies every page longhand (full training on raw data); the other skims solved examples, drills the hard exercises, and skips the misprints (masking plus filtering). Exam day favors the skimmer — same syllabus, quarter of the hours, sharper on what matters.

Efficient pretraining is disciplined skimming. Masking skips the predictable patches; filtering skips the misprint images; staging reads the easy chapters first. Copyists object that skimmers "see less" — true, and beside the point, because seeing wisely beats seeing everything.

How It Actually Works

Recipes layer three savings. Mask: $75\%$ patch masking with tiny decoders (asymmetric MAE design: big encoder on visible tokens, small decoder for reconstruction). Filter: score pools with cheap models, keep $20$ – $40\%$ (CLIP-score, aesthetic, NSFW and dedup screens). Stage: warm up at low resolution, finish short at full — most learning happens coarse. Mixed precision, gradient checkpointing, and sharded optimizers multiply the savings underneath.

Measure wall-clock per quality point (linear-probe accuracy per GPU-hour), not epochs: efficient methods win some epochs by losing others.

Worked example

Pool $400$ M images; keep $30\%$ after filtering: $120$ M. Tokens per image $196$ ; visible after $75\%$ masking: $49$ ( $25\%$ ). Attention work scales roughly with token fraction squared per image times image fraction: $0.30 \times 0.25 \approx 0.075$ of the full-data full-token bill — over $13\times$ cheaper ( $1/0.075 \approx 13.3$ ).

Linear costs (decoding, I/O) scale nearer $0.30 \times 0.25$ as well, so the whole run lands roughly an order of magnitude cheaper. That order of magnitude is the difference between a corporate cluster and a university one.

Code

The point of this snippet is the multiplication: filtering times masking sets the budget before a single GPU spins.

1def budget_fraction(keep_images: float, keep_tokens: float) -> float:2    return keep_images * keep_tokens  # independent savings multiply3
4
5print(round(budget_fraction(0.3, 0.25), 4))  # -> 0.0756print(round(1 / 0.075, 1))  # -> 13.3

Watch Out For

Filters that bake in their own blindness

Quality scorers have tastes: aesthetic models downvote dark-skinned faces and night scenes, text-alignment filters drop non-English content, dedup erases rare cases as "duplicates". Filtering to $30\%$ with a biased scorer manufactures a biased dataset wearing efficiency's halo.

Audit kept-versus-dropped slices for demographic and content skew before training. Efficiency that homogenizes the data charges interest at deployment — paid by whoever the filter disliked.

Masking ratios tuned for someone else's modality

Vision's $75\%$ sweet spot assumes image redundancy; video (more redundant) wants $90\%$ , text (less) wants $15\%$ . Copying ratios across modalities under-trains dense signals and overprices sparse ones.

Sweep masking per modality and report the curve, not just the winner. The ratio is a property of the data's redundancy, never a universal constant.

The Quick Version

Efficient pretraining multiplies three savings: masking, filtering, and staged resolutions.
The worked fixture keeps $30\%$ of images and $25\%$ of tokens: $0.075$ of the bill, $13.3\times$ cheaper.
Measure accuracy per GPU-hour on target hardware, not epochs or FLOPs in a vacuum.
Audit filters for skew: efficiency must not launder bias into the dataset.
Sweep masking per modality; redundancy sets the ratio, not convention.