Skip to content
AI360XpertBeta

Compute-Optimal Training

For a fixed training budget, the smartest model splits spending evenly between size and data. A giant model starved of tokens loses to a smaller one trained on far more text.

At a fixed training budget, a balanced split between parameters and tokens reaches lower loss than a lopsided split with the same FLOPs
At a fixed training budget, a balanced split between parameters and tokens reaches lower loss than a lopsided split with the same FLOPs

Why Does This Exist?

You get one pretraining budget. Every FLOP you spend making the model bigger is a FLOP you cannot spend showing it more text, and early teams consistently guessed this split wrong. They built very large models on modest token counts, then paid for that choice twice: once in training, and again every day in serving, because a bloated model costs more on every single request.

The fix is a budgeting rule. Compute-optimal training tells you how to divide a fixed FLOP budget between model size and data size so the final loss is as low as it can go. The short answer is that the two grow together, in roughly equal proportion.

This page covers that pretraining split only. The objective those tokens train against is next-token prediction. What actually fills the token budget, code versus math versus prose, lives in data mixing. What you do after pretraining lives in pages like full fine-tuning.

The one prerequisite fits in two sentences. Let NN be your parameter count, meaning the number of trainable weights in the model. Let DD be your training token count, meaning the number of tokens the model sees during training, and let CC be total training compute in FLOPs.

Think of It Like This

A catering budget: a bigger oven or more ingredients

You have a fixed catering budget for a wedding. You can rent a huge professional oven and buy a tiny bag of groceries, or rent a small oven and buy a mountain of groceries. The huge oven with almost nothing to cook gives you burnt scraps. The mountain of groceries with a toy oven gives you a queue that never gets fed.

The best wedding comes from the middle: an oven matched to the amount of food. Parameters are your oven capacity and tokens are your ingredients. Doubling the oven while keeping groceries flat does not double the meal.

Where it stops: an oven never learns, while a model does, so the analogy says nothing about data quality or ordering. It also says nothing about serving cost after the wedding, which is exactly where the training-optimal answer can mislead you.

How It Actually Works

Training compute follows one simple relation. For a dense transformer, the cost is approximately

C≈6NDC \approx 6ND

In plain words: CC is training FLOPs, NN is parameter count, DD is token count. The 6 comes from about 2ND2ND for the forward pass plus about 4ND4ND for the backward pass on every token. This is an approximation that ignores attention extras and hardware overhead, but it is the one everyone budgets with.

Why even splits win

Loss falls when you add parameters and falls when you add tokens, but each one bottlenecks the other. A huge NN with tiny DD memorizes instead of generalizing. A tiny NN with huge DD runs out of capacity and its curve flattens no matter how much text you pour in.

So at fixed CC, the lowest loss sits where neither side starves. Fitting loss curves across hundreds of training runs gives the practical rule D≈20ND \approx 20N, about 20 training tokens per parameter. If you scale the budget by 10, you scale each side by about 3.16, since 10≈3.16×3.1610 \approx 3.16 \times 3.16. That keeps the ratio while spending the full 10.

Worked example: one budget, two models

Take a fixed budget of C=1.2×1020C = 1.2 \times 10^{20} FLOPs. With the rule D=20ND = 20N, substitute into C≈6NDC \approx 6ND:

C≈6N(20N)=120N2C \approx 6N(20N) = 120N^2

Solve for NN:

N=C/120=1.2×1020/120=1.0×1018=1.0×109N = \sqrt{C/120} = \sqrt{1.2 \times 10^{20}/120} = \sqrt{1.0 \times 10^{18}} = 1.0 \times 10^9

So the balanced model is N=1N = 1 billion parameters with D=20×1D = 20 \times 1 billion =20= 20 billion tokens. Check it: 6×109×20×109=120×1018=1.2×10206 \times 10^9 \times 20 \times 10^9 = 120 \times 10^{18} = 1.2 \times 10^{20}. It spends the whole budget exactly.

Now spend that same budget lopsided: N=4N = 4 billion parameters with only D=5D = 5 billion tokens. Check: 6×4×109×5×109=120×1018=1.2×10206 \times 4 \times 10^9 \times 5 \times 10^9 = 120 \times 10^{18} = 1.2 \times 10^{20}. Same FLOPs, but the ratio D/ND/N is 1.251.25 instead of 2020. The big model sees each token pattern far fewer times per parameter, memorizes the small set, and finishes with higher loss. You paid the same and got less, plus a model four times larger to serve.

When training-optimal is the wrong target

The rule above minimizes training FLOPs for a target loss. It ignores serving. A model trains once and serves millions of times, so teams often deliberately overtrain: they pick a smaller NN than the rule allows and train it far past D=20ND = 20N.

That costs extra training FLOPs on purpose. What it buys is a small, sharp model that is cheap and fast on every later request. If you expect heavy deployment, minimize the serving bill, not the training bill. The two optima are different questions, so answer which bill you care about before you split the budget.

Code

This snippet computes the balanced split for any budget, using the same algebra as the worked example. It takes a FLOP count and returns the matching parameter and token counts.

import math

def optimal_split(compute_flops: float) -> tuple[float, float]:    """Return (params, tokens) for a compute optimal split with D = 20N."""    params = math.sqrt(compute_flops / 120.0)  # from C = 6*N*20N = 120*N^2    tokens = 20.0 * params    return params, tokens

n, d = optimal_split(1.2e20)print(f"{n / 1e9:.1f}B params")   # -> 1.0B paramsprint(f"{d / 1e9:.1f}B tokens")   # -> 20.0B tokensprint(f"{d / n:.1f}x")            # -> 20.0x

It prints 11 billion parameters and 2020 billion tokens, the balanced point from the worked example. Change the input and the ratio stays 2020, while each side grows with the square root of the budget.

Watch Out For

Treating 20 tokens per parameter as a law of nature

The symptom: you hit the ratio exactly and the loss still stalls, so you assume the scaling laws are broken. The cause is usually the assumption underneath the number. The 2020 figure was fit on dense transformers with a particular data mix and quality level. Cleaner or more repetitive data, a different tokenizer, or a mixture-of-experts architecture moves the best point.

Fix it by treating 2020 as your starting split, not your answer. Hold out loss is the judge. If adding tokens past the ratio keeps dropping held-out loss faster than adding parameters, keep feeding tokens and re-fit your own split.

Minimizing the training bill while ignoring the serving bill

The symptom: your run was compute-optimal on paper, yet the model is too large to deploy cheaply and latency budgets break. The cause is optimizing the wrong cost. Training-optimal answers how to spend CC once. It never asks what each future request will cost in memory and latency.

Fix it before you train: estimate request volume first. If you will serve heavily, pick a smaller NN on purpose and overtrain it past the Chinchilla point, the way current small deployment-focused models do. Say which optimum you chose in your run notes so nobody mistakes it for a training-optimal split later.

The Quick Version

  • At a fixed training budget, the lowest loss comes from growing parameters and tokens together, not from maxing one side.
  • Training cost follows C approx 6ND, so a 1B parameter model paired with 20B tokens costs about 1.2e20 FLOPs.
  • The practical rule is about 20 training tokens per parameter, fitted from runs that compared many splits at equal compute.
  • A 4B model trained on only 5B tokens spends the same FLOPs as the balanced 1B on 20B setup and finishes worse.
  • Overtraining a small model past the ratio wastes training FLOPs on purpose to buy cheaper serving on every later request.
  • The 20x figure assumes dense transformers and a fixed data mix, so re-check it when data quality or architecture changes.