Skip to content
AI360XpertBeta

Scaling Laws

Double compute, data, and model size together and test loss falls along a smooth predictable curve. That curve lets you forecast a huge training run from small cheap ones.

Balanced growth in compute, data, and size holds test loss on a straight log-log line, while growing one input alone makes the curve flatten
Balanced growth in compute, data, and size holds test loss on a straight log-log line, while growing one input alone makes the curve flatten

Why Does This Exist?

You would not rent thousands of GPUs for months on a hunch. Before you pay, you want a straight answer: if we spend twice as much, what loss do we get? Scaling laws give you that forecast. They turn a handful of small runs into a line you can read forward.

The pattern is plain. Grow model size, training data, and compute together and average test loss falls along a smooth curve. Plot it with both axes on a log scale and the curve straightens into a line. Fit that line through cheap points and you can read off what a far larger run should reach, provided you keep the recipe fixed. That is the whole bet behind modern training budgets.

This page covers that curve only: what it claims, why the log view straightens it, and where it stops holding. The split between size and data lives in compute-optimal training. What the average hides on any single task lives in emergent abilities.

One idea first, because everything below rests on it. Next-token prediction grades the model on each next chunk of text and averages those grades into test loss. Perplexity reports the same number in different units. When this page says loss falls, it means that average.

Think of It Like This

Training for a 5K with miles, legs, and sleep

You decide to run a faster 5K. Three inputs matter: weekly miles, leg strength, and sleep. You double all three together for two months and your times drop on schedule. More work in, faster times out. So far, so predictable.

Now try a different plan. Triple your miles but keep sleeping four hours a night and never touch a weight. Times stall, then slide backward. You did not lack effort. You starved two inputs while flooding the third, and the weakest one set the result. That stall is exactly what happens when a team scales parameters while reusing a small tired dataset, or pours in data a tiny model cannot absorb.

Where it stops: your body gets hurt and needs rest, while weights just sit there. And race day is one windy morning, while test loss averages millions of guesses. The analogy teaches joint growth and the stall. It says nothing about the math that makes the line straight.

How It Actually Works

Name the inputs. Let NN be parameter count, meaning the number of trainable weights. Let DD be dataset size, meaning the number of training tokens the model sees. Let CC be compute, meaning total training FLOPs. Let LL be average test loss, meaning the mean next-token error on held-out text. Lower LL is better.

When the other inputs grow alongside it, loss against any single input XX follows a power law shape:

L(X)=L∞+(X0X)αL(X) = L_{\infty} + \left(\frac{X_0}{X}\right)^{\alpha}

In plain words: L∞L_{\infty} (L infinity) is the floor, the part even endless scale cannot remove. X0X_0 (X nought) is a scale constant set by your model family and data mix. The exponent α\alpha (alpha) is the slope, a small positive number you fit from runs. The bracket term is the reducible part, the error you can still sand away. Doubling XX always shrinks that term by the same factor, 2−α2^{-\alpha}, which is why each doubling buys a similar drop.

Take logs and the straight line appears. Subtract the floor, then log both sides:

log⁡(L−L∞)=αlog⁡X0−αlog⁡X\log\left(L - L_{\infty}\right) = \alpha \log X_0 - \alpha \log X

Read it as: log reducible loss equals a constant minus α\alpha times log input. On axes where both scales are logarithmic, that is a line with slope −α-\alpha. This is the picture above. The focal solid line is balanced growth. The dashed line is what happens when you push one input while the others sit still.

Why balanced growth stays on the line

Each input bottlenecks the rest. A huge NN with tiny DD memorizes the small set instead of generalizing, so held-out loss stalls. A tiny NN with huge DD runs out of capacity, so extra tokens wash past it. Only when NN and DD rise together does CC buy its full drop. The practical split between the two is its own budgeting question, worked through in compute-optimal training. Pretraining is where most teams first meet this trade, since the corpus and the model meet there.

The line also assumes a fixed recipe: same architecture family, same data mix and quality, same training setup, and mostly fresh tokens. Change any of those and you draw a new line with a new slope. That is normal. The claim is not one universal line. It is that each fixed recipe gives a line you can extend.

What the average buys you, and what it hides

The trade is sharp. Averaging over mountains of text buys forecastability: the mean moves smoothly even when single examples swing wildly. What it costs is resolution. Your product rarely asks for a good average. It asks for five correct steps in a row, and it grades near misses as failures. That task score can sit near zero while the average glides down, then climb steeply once every step starts landing. Do not read a smooth loss curve as a promise about your feature. Read emergent abilities for the jump, and measuring general capability before you turn any of this into a launch date.

So when do you not use scaling laws? Do not use them to predict one narrow skill, to justify training past a data wall of repeats and sludge, or to pick training size without pricing serving. A model trains once and answers millions of times. If serving dominates your bill, the training-optimal point is the wrong target, and test-time compute may buy more quality per coin than a bigger pretraining run.

Worked example with three budgets on one line

Treat these numbers as an illustration you can check by hand, not a fitted result from any lab. Real exponents come from fitting your own runs. Set the floor L∞=1.70L_{\infty} = 1.70, the scale X0=1X_0 = 1 compute unit, and the slope α=0.15\alpha = 0.15. Measure compute in arbitrary units at C=1C = 1, 88, and 6464, each step an 8x jump so the log steps are equal.

Reducible loss is (1/C)0.15(1/C)^{0.15}, which is C−0.15C^{-0.15} in plain symbols:

  • At C=1C = 1: 1−0.15=1.0001^{-0.15} = 1.000. Total L=1.70+1.000=2.70L = 1.70 + 1.000 = 2.70.
  • At C=8C = 8: 8−0.15=exp⁡(−0.15ln⁡8)=exp⁡(−0.3119)≈0.7328^{-0.15} = \exp(-0.15 \ln 8) = \exp(-0.3119) \approx 0.732. Total L=1.70+0.732=2.43L = 1.70 + 0.732 = 2.43.
  • At C=64C = 64: 64−0.15=exp⁡(−0.15ln⁡64)=exp⁡(−0.6238)≈0.53664^{-0.15} = \exp(-0.15 \ln 64) = \exp(-0.6238) \approx 0.536. Total L=1.70+0.536=2.24L = 1.70 + 0.536 = 2.24.

Each 8x of compute shaves less in absolute terms, 0.270.27 then 0.190.19, which is diminishing returns doing its job. In logs the steps are exactly even: log⁡10C\log_{10} C runs 00, 0.9030.903, 1.8061.806, while log⁡10\log_{10} reducible runs 00, −0.135-0.135, −0.271-0.271. Same input step, same log drop. That even spacing is the straight line drawn above, with the three totals 2.702.70, 2.432.43, and 2.242.24 sitting beside its dots. The dashed curve beside it uses the same start but flattens, which is the single-axis stall drawn for contrast.

Code

This snippet replays the worked example. It takes a compute value in the same arbitrary units and returns the illustrative total loss, then prints the three points from the diagram.

import math

def total_loss(compute: float, floor: float = 1.70, alpha: float = 0.15) -> float:    """Illustrative power law loss for compute in arbitrary units."""    reducible = (1.0 / compute) ** alpha  # (X0 / X)^alpha with X0 = 1    return floor + reducible

for c in (1.0, 8.0, 64.0):    print(f"{c:.0f} -> {total_loss(c):.2f}")    # -> 1 -> 2.70    # -> 8 -> 2.43    # -> 64 -> 2.24

It prints 2.702.70, 2.432.43, and 2.242.24, the three hand-computed totals. Change alpha and watch the slope move: larger alpha steepens the line, smaller alpha flattens it. That slope is what real teams fit from their own small runs instead of borrowing from here.

Watch Out For

Promising a capability because the average fell

Symptom: loss lands exactly on the forecast, yet your multi-step task still scores near zero, and the roadmap already promised it for next quarter. The team concludes scaling failed, or ships a feature that cannot do its one job.

The cause is reading the wrong number. Loss averages millions of easy and hard guesses, while your feature demands every step at once. Fix it by tracking the task metric next to the average from the start, with a continuous per-step score alongside strict exact match. If the average glides while the task sleeps, the direction may still be working. If both sleep, change direction. Check benchmark contamination before you celebrate any sudden jump.

Scaling one axis and calling the stall a broken law

Symptom: you quadruple parameters on the same token pile, or pour tokens into a tiny model, and loss flattens well above the line. It looks like the law quit on you.

The cause is a bottleneck you built. Reused or low-quality text stops teaching, and a small model stops absorbing. Fix it before the next run: grow NN and DD together per compute-optimal training, deduplicate against evals, and hold out data quality as a variable instead of assuming it. If the curve still flattens, you found your current wall. Believe it.

The Quick Version

  • Scaling laws say average test loss falls as a power law in compute, data, and size when the three grow together.
  • On log-log axes the reducible part is a straight line, so small runs forecast large ones made the same way.
  • An 8x compute jump with floor 1.70 and slope 0.15 moves illustrative loss 2.70 to 2.43 to 2.24, with even log steps of 0.135.
  • Growing one input while starving the rest flattens the curve, through memorization on one side and missing capacity on the other.
  • The average forecasts the mean and hides single tasks, so track task scores separately before promising anything.
  • The line assumes a fixed recipe and fresh quality data, so a new mix or repeated sludge draws a new line.