Scaling Laws
Double compute, data, and model size together and test loss falls along a smooth predictable curve. That curve lets you forecast a huge training run from small cheap ones.
Why Does This Exist?
You would not rent thousands of GPUs for months on a hunch. Before you pay, you want a straight answer: if we spend twice as much, what loss do we get? Scaling laws give you that forecast. They turn a handful of small runs into a line you can read forward.
The pattern is plain. Grow model size, training data, and compute together and average test loss falls along a smooth curve. Plot it with both axes on a log scale and the curve straightens into a line. Fit that line through cheap points and you can read off what a far larger run should reach, provided you keep the recipe fixed. That is the whole bet behind modern training budgets.
This page covers that curve only: what it claims, why the log view straightens it, and where it stops holding. The split between size and data lives in compute-optimal training. What the average hides on any single task lives in emergent abilities.
One idea first, because everything below rests on it. Next-token prediction grades the model on each next chunk of text and averages those grades into test loss. Perplexity reports the same number in different units. When this page says loss falls, it means that average.
Think of It Like This
Training for a 5K with miles, legs, and sleep
You decide to run a faster 5K. Three inputs matter: weekly miles, leg strength, and sleep. You double all three together for two months and your times drop on schedule. More work in, faster times out. So far, so predictable.
Now try a different plan. Triple your miles but keep sleeping four hours a night and never touch a weight. Times stall, then slide backward. You did not lack effort. You starved two inputs while flooding the third, and the weakest one set the result. That stall is exactly what happens when a team scales parameters while reusing a small tired dataset, or pours in data a tiny model cannot absorb.
Where it stops: your body gets hurt and needs rest, while weights just sit there. And race day is one windy morning, while test loss averages millions of guesses. The analogy teaches joint growth and the stall. It says nothing about the math that makes the line straight.
How It Actually Works
Name the inputs. Let be parameter count, meaning the number of trainable weights. Let be dataset size, meaning the number of training tokens the model sees. Let be compute, meaning total training FLOPs. Let be average test loss, meaning the mean next-token error on held-out text. Lower is better.
When the other inputs grow alongside it, loss against any single input follows a power law shape:
In plain words: (L infinity) is the floor, the part even endless scale cannot remove. (X nought) is a scale constant set by your model family and data mix. The exponent (alpha) is the slope, a small positive number you fit from runs. The bracket term is the reducible part, the error you can still sand away. Doubling always shrinks that term by the same factor, , which is why each doubling buys a similar drop.
Take logs and the straight line appears. Subtract the floor, then log both sides:
Read it as: log reducible loss equals a constant minus times log input. On axes where both scales are logarithmic, that is a line with slope . This is the picture above. The focal solid line is balanced growth. The dashed line is what happens when you push one input while the others sit still.
Why balanced growth stays on the line
Each input bottlenecks the rest. A huge with tiny memorizes the small set instead of generalizing, so held-out loss stalls. A tiny with huge runs out of capacity, so extra tokens wash past it. Only when and rise together does buy its full drop. The practical split between the two is its own budgeting question, worked through in compute-optimal training. Pretraining is where most teams first meet this trade, since the corpus and the model meet there.
The line also assumes a fixed recipe: same architecture family, same data mix and quality, same training setup, and mostly fresh tokens. Change any of those and you draw a new line with a new slope. That is normal. The claim is not one universal line. It is that each fixed recipe gives a line you can extend.
What the average buys you, and what it hides
The trade is sharp. Averaging over mountains of text buys forecastability: the mean moves smoothly even when single examples swing wildly. What it costs is resolution. Your product rarely asks for a good average. It asks for five correct steps in a row, and it grades near misses as failures. That task score can sit near zero while the average glides down, then climb steeply once every step starts landing. Do not read a smooth loss curve as a promise about your feature. Read emergent abilities for the jump, and measuring general capability before you turn any of this into a launch date.
So when do you not use scaling laws? Do not use them to predict one narrow skill, to justify training past a data wall of repeats and sludge, or to pick training size without pricing serving. A model trains once and answers millions of times. If serving dominates your bill, the training-optimal point is the wrong target, and test-time compute may buy more quality per coin than a bigger pretraining run.
Worked example with three budgets on one line
Treat these numbers as an illustration you can check by hand, not a fitted result from any lab. Real exponents come from fitting your own runs. Set the floor , the scale compute unit, and the slope . Measure compute in arbitrary units at , , and , each step an 8x jump so the log steps are equal.
Reducible loss is , which is in plain symbols:
- At : . Total .
- At : . Total .
- At : . Total .
Each 8x of compute shaves less in absolute terms, then , which is diminishing returns doing its job. In logs the steps are exactly even: runs , , , while reducible runs , , . Same input step, same log drop. That even spacing is the straight line drawn above, with the three totals , , and sitting beside its dots. The dashed curve beside it uses the same start but flattens, which is the single-axis stall drawn for contrast.
Code
This snippet replays the worked example. It takes a compute value in the same arbitrary units and returns the illustrative total loss, then prints the three points from the diagram.
import math
def total_loss(compute: float, floor: float = 1.70, alpha: float = 0.15) -> float: """Illustrative power law loss for compute in arbitrary units.""" reducible = (1.0 / compute) ** alpha # (X0 / X)^alpha with X0 = 1 return floor + reducible
for c in (1.0, 8.0, 64.0): print(f"{c:.0f} -> {total_loss(c):.2f}") # -> 1 -> 2.70 # -> 8 -> 2.43 # -> 64 -> 2.24It prints , , and , the three hand-computed totals. Change alpha and watch the slope move: larger alpha steepens the line, smaller alpha flattens it. That slope is what real teams fit from their own small runs instead of borrowing from here.
Watch Out For
Promising a capability because the average fell
Symptom: loss lands exactly on the forecast, yet your multi-step task still scores near zero, and the roadmap already promised it for next quarter. The team concludes scaling failed, or ships a feature that cannot do its one job.
The cause is reading the wrong number. Loss averages millions of easy and hard guesses, while your feature demands every step at once. Fix it by tracking the task metric next to the average from the start, with a continuous per-step score alongside strict exact match. If the average glides while the task sleeps, the direction may still be working. If both sleep, change direction. Check benchmark contamination before you celebrate any sudden jump.
Scaling one axis and calling the stall a broken law
Symptom: you quadruple parameters on the same token pile, or pour tokens into a tiny model, and loss flattens well above the line. It looks like the law quit on you.
The cause is a bottleneck you built. Reused or low-quality text stops teaching, and a small model stops absorbing. Fix it before the next run: grow and together per compute-optimal training, deduplicate against evals, and hold out data quality as a variable instead of assuming it. If the curve still flattens, you found your current wall. Believe it.
The Quick Version
- Scaling laws say average test loss falls as a power law in compute, data, and size when the three grow together.
- On log-log axes the reducible part is a straight line, so small runs forecast large ones made the same way.
- An 8x compute jump with floor 1.70 and slope 0.15 moves illustrative loss 2.70 to 2.43 to 2.24, with even log steps of 0.135.
- Growing one input while starving the rest flattens the curve, through memorization on one side and missing capacity on the other.
- The average forecasts the mean and hides single tasks, so track task scores separately before promising anything.
- The line assumes a fixed recipe and fresh quality data, so a new mix or repeated sludge draws a new line.