Data Mixing and Curriculum
You set the exact share of web, code, and math in every batch instead of accepting whatever the crawl contains. Then you shift those shares from easy to hard as training runs.
Why Does This Exist?
If you train on the web as you find it, you get the web as it is. Mostly filler, with the good stuff buried. Code and math are a thin slice of any crawl, yet they carry most of the reasoning signal. Train on raw proportions and your model reads everything and learns little logic. It writes like a forum.
So you stop accepting the pile and start setting shares. Data mixing fixes the fraction of each batch that comes from each domain. Curriculum moves those fractions over time, easy first and dense later. You'll spend most of your effort on the first and most of your regret on the second, because the schedule is where forgetting sneaks in.
This page covers the shares, the shift, and the interleaving that keeps old skills alive. The run they feed is pretraining: predicting the next token across a huge corpus, where the mix decides what gets predicted most.
Think of It Like This
A canteen menu and a term plan
Your school pantry is 80 percent pasta, because pasta is cheap and keeps. You don't serve 80 percent pasta. You write a menu: about half grains, a good share of protein, a solid share of greens, on every plate, every day. That menu is the mix. Raw availability never decides the meal.
Then you sequence the term. Early weeks: reading, arithmetic, simple sentences. Later: essays, algebra, lab reports. Same kitchen, shifting shares. That's the curriculum.
Where it stops: pupils don't memorize the menu by repetition, and they keep old subjects warm with homework. Your model does neither on its own. Repeat a small math pool four times per pass and it starts reciting instead of reasoning. Drop prose to zero for a month and grammar sags while code climbs. So you cap repeats, and you never serve zero of anything.
How It Actually Works
Call each source a domain : web, code, math. Write for its raw share on disk, meaning what the crawl gives you, and for its sampling weight, meaning what you choose. Weights are probabilities, so they sum to one: . For a batch of tokens, where is the batch size in tokens, the expected count from domain is , where is that domain's expected tokens in the batch. The upsampling factor , where says how much harder you lean on a domain than nature does, tells you the price. Above one means repeated. Below one means throttled.
Worked example: one batch of 1,000 tokens
Take a toy batch with . Raw shares are for web, code, and math. You set . Expected counts fall straight out:
Check the sum: . Now the factors: web , code , math . Each math token shows up about four times per pass on average. That single number is the whole trick and the whole risk.
Shift the shares as the model matures
Early on the model needs breadth: grammar, facts, style. Late, it needs density: proofs, programs, derivations. So you schedule phases. Start at , which from 1,000 tokens gives . End at , which gives . Same batch size, same three domains, different diet. The arrow in the diagram is that move.
Two neighbours help with the next decisions. When the question is how many tokens to buy at all, read compute-optimal training. When the new tokens will be machine-written rather than crawled, read synthetic data for post-training.
Why you must interleave every domain
Train French for a month, then Python for a month, and the French is gone. Gradients from the new domain overwrite the weights the old one needed. That is catastrophic forgetting: lost skill from sequential training, and the fix is boring. Never drop a domain to zero. Every batch carries every domain, so each update tugs the weights toward values that serve all of them at once.
The price is permanent tension. Push math higher and reasoning climbs while memorization risk and lost web coverage climb with it. There is no share that wins everywhere. There is only the share whose failure you can live with, measured per domain, not on average.
Code
This check turns weights into counts and flags repeats before any long run. Run it whenever the mix changes.
def expected_counts(batch_size: int, weights: list[float]) -> list[int]: return [round(batch_size * w) for w in weights]
def upsampling(weights: list[float], raw: list[float]) -> list[float]: return [round(w / r, 3) for w, r in zip(weights, raw)]
batch = 1000w = [0.50, 0.30, 0.20]r = [0.80, 0.15, 0.05]print(expected_counts(batch, w)) # -> [500, 300, 200]print(upsampling(w, r)) # -> [0.625, 2.0, 4.0]The first line is your batch. The second is your repeat warning. Math at 4.0 means four looks per pass: fine for a stretch, memorization if you hold it. The same two functions still describe each batch once training moves on to supervised fine-tuning.
Watch Out For
Repeating a small pool until it recites
Math looks strong in training, then collapses on reworded problems. The symptom is specific: the model quotes training documents nearly verbatim, its loss on that domain keeps falling, and held-out loss on the same domain stalls or rises. You've crossed from learning into memorizing.
Cap exposures per document before the run, dedupe the pool, and hold out canary documents you never train on. Watch the train minus held-out gap per domain, not the average loss. When the math gap opens, lower its weight instead of training longer.
Zeroing a domain and forgetting it
A code-heavy phase works: programming scores climb. Then you notice grammar slipping, facts going stale, style flattening. Per-domain evals show the old domain sliding week by week while the new one shines. That's forgetting, not noise.
Give every domain a floor above zero in every phase and put every domain in every batch. Track per-domain loss separately and end the phase when the old domain turns down, even if the new one still climbs.
The Quick Version
- Data mixing sets the sampling probability per domain, so each batch holds what you chose instead of what the crawl contained.
- With weights 50/30/20 over 1,000 tokens, expect 500 web, 300 code, and 200 math tokens, with each math token repeating about 4 times per pass.
- A curriculum starts broad near 65/25/10 and ends dense near 35/35/30, keeping the batch size fixed while the diet changes.
- Every batch must carry every domain, because training one domain at a time overwrites the skills the last one needed.
- Upsampling buys reasoning and sells memorization: cap repeats, dedupe small pools, and watch held-out loss separately for each domain.