Compute-Optimal Training
For a fixed training budget, the smartest model splits spending evenly between size and data. A giant model starved of tokens loses to a smaller one trained on far more text.
Why Does This Exist?
You get one pretraining budget. Every FLOP you spend making the model bigger is a FLOP you cannot spend showing it more text, and early teams consistently guessed this split wrong. They built very large models on modest token counts, then paid for that choice twice: once in training, and again every day in serving, because a bloated model costs more on every single request.
The fix is a budgeting rule. Compute-optimal training tells you how to divide a fixed FLOP budget between model size and data size so the final loss is as low as it can go. The short answer is that the two grow together, in roughly equal proportion.
This page covers that pretraining split only. The objective those tokens train against is next-token prediction. What actually fills the token budget, code versus math versus prose, lives in data mixing. What you do after pretraining lives in pages like full fine-tuning.
The one prerequisite fits in two sentences. Let be your parameter count, meaning the number of trainable weights in the model. Let be your training token count, meaning the number of tokens the model sees during training, and let be total training compute in FLOPs.
Think of It Like This
A catering budget: a bigger oven or more ingredients
You have a fixed catering budget for a wedding. You can rent a huge professional oven and buy a tiny bag of groceries, or rent a small oven and buy a mountain of groceries. The huge oven with almost nothing to cook gives you burnt scraps. The mountain of groceries with a toy oven gives you a queue that never gets fed.
The best wedding comes from the middle: an oven matched to the amount of food. Parameters are your oven capacity and tokens are your ingredients. Doubling the oven while keeping groceries flat does not double the meal.
Where it stops: an oven never learns, while a model does, so the analogy says nothing about data quality or ordering. It also says nothing about serving cost after the wedding, which is exactly where the training-optimal answer can mislead you.
How It Actually Works
Training compute follows one simple relation. For a dense transformer, the cost is approximately
In plain words: is training FLOPs, is parameter count, is token count. The 6 comes from about for the forward pass plus about for the backward pass on every token. This is an approximation that ignores attention extras and hardware overhead, but it is the one everyone budgets with.
Why even splits win
Loss falls when you add parameters and falls when you add tokens, but each one bottlenecks the other. A huge with tiny memorizes instead of generalizing. A tiny with huge runs out of capacity and its curve flattens no matter how much text you pour in.
So at fixed , the lowest loss sits where neither side starves. Fitting loss curves across hundreds of training runs gives the practical rule , about 20 training tokens per parameter. If you scale the budget by 10, you scale each side by about 3.16, since . That keeps the ratio while spending the full 10.
Worked example: one budget, two models
Take a fixed budget of FLOPs. With the rule , substitute into :
Solve for :
So the balanced model is billion parameters with billion billion tokens. Check it: . It spends the whole budget exactly.
Now spend that same budget lopsided: billion parameters with only billion tokens. Check: . Same FLOPs, but the ratio is instead of . The big model sees each token pattern far fewer times per parameter, memorizes the small set, and finishes with higher loss. You paid the same and got less, plus a model four times larger to serve.
When training-optimal is the wrong target
The rule above minimizes training FLOPs for a target loss. It ignores serving. A model trains once and serves millions of times, so teams often deliberately overtrain: they pick a smaller than the rule allows and train it far past .
That costs extra training FLOPs on purpose. What it buys is a small, sharp model that is cheap and fast on every later request. If you expect heavy deployment, minimize the serving bill, not the training bill. The two optima are different questions, so answer which bill you care about before you split the budget.
Code
This snippet computes the balanced split for any budget, using the same algebra as the worked example. It takes a FLOP count and returns the matching parameter and token counts.
import math
def optimal_split(compute_flops: float) -> tuple[float, float]: """Return (params, tokens) for a compute optimal split with D = 20N.""" params = math.sqrt(compute_flops / 120.0) # from C = 6*N*20N = 120*N^2 tokens = 20.0 * params return params, tokens
n, d = optimal_split(1.2e20)print(f"{n / 1e9:.1f}B params") # -> 1.0B paramsprint(f"{d / 1e9:.1f}B tokens") # -> 20.0B tokensprint(f"{d / n:.1f}x") # -> 20.0xIt prints billion parameters and billion tokens, the balanced point from the worked example. Change the input and the ratio stays , while each side grows with the square root of the budget.
Watch Out For
Treating 20 tokens per parameter as a law of nature
The symptom: you hit the ratio exactly and the loss still stalls, so you assume the scaling laws are broken. The cause is usually the assumption underneath the number. The figure was fit on dense transformers with a particular data mix and quality level. Cleaner or more repetitive data, a different tokenizer, or a mixture-of-experts architecture moves the best point.
Fix it by treating as your starting split, not your answer. Hold out loss is the judge. If adding tokens past the ratio keeps dropping held-out loss faster than adding parameters, keep feeding tokens and re-fit your own split.
Minimizing the training bill while ignoring the serving bill
The symptom: your run was compute-optimal on paper, yet the model is too large to deploy cheaply and latency budgets break. The cause is optimizing the wrong cost. Training-optimal answers how to spend once. It never asks what each future request will cost in memory and latency.
Fix it before you train: estimate request volume first. If you will serve heavily, pick a smaller on purpose and overtrain it past the Chinchilla point, the way current small deployment-focused models do. Say which optimum you chose in your run notes so nobody mistakes it for a training-optimal split later.
The Quick Version
- At a fixed training budget, the lowest loss comes from growing parameters and tokens together, not from maxing one side.
- Training cost follows C approx 6ND, so a 1B parameter model paired with 20B tokens costs about 1.2e20 FLOPs.
- The practical rule is about 20 training tokens per parameter, fitted from runs that compared many splits at equal compute.
- A 4B model trained on only 5B tokens spends the same FLOPs as the balanced 1B on 20B setup and finishes worse.
- Overtraining a small model past the ratio wastes training FLOPs on purpose to buy cheaper serving on every later request.
- The 20x figure assumes dense transformers and a fixed data mix, so re-check it when data quality or architecture changes.