Skip to content
AI360XpertBeta

Emergent Abilities

Some model skills look absent until scale crosses a threshold, then they switch on fast. The jump is usually in how you score the task, not a sudden rewiring inside the model.

Smooth gains in per-step accuracy hide beneath sharp jumps in exact task success until every step clicks at larger scale
Smooth gains in per-step accuracy hide beneath sharp jumps in exact task success until every step clicks at larger scale

Why Does This Exist?

You can watch average loss fall smoothly as you add compute, then ship a model that still scores zero on the one task you care about. That gap is why this page exists.

Scaling laws describe the smooth part: mean error over mountains of text, improving in a predictable curve as models and data grow. Your product rarely asks for a good average. It asks for five correct steps in a row, or a fully working answer, and it grades anything short of that as failure.

So you get two pictures of the same training run. The average keeps improving. The task score sits near zero, then climbs steeply once the model is strong enough to land every step together. This page explains that second picture. For the smooth curve itself, read scaling laws. For how to test broad skills without fooling yourself, read measuring general capability.

One idea first, because everything below rests on it. Next-token prediction trains the model to guess each next chunk of text, and loss averages those guesses. Task success does not average. It demands the whole chain, and that difference creates the jump.

Think of It Like This

A combination lock with five dials

Suppose your front door has five dials and opens only when all five sit on the right number. You get better at setting each dial week by week. A friend watching only the door sees nothing for weeks: locked, locked, locked. Then one week every dial lands and the door swings open. From the door's point of view your skill jumped from zero to perfect overnight. From your hands' point of view it improved a little each week.

Map it across. Each dial is one step of the task. Your steady hands are per-step accuracy. The door is exact match scoring: all steps right or it counts as wrong. Practice is scale.

Where it stops: you learn each dial separately, while a model shares one set of weights across all steps, so its steps rise and fall together. And a real door gives no partial credit by design, while you should always measure partial progress during evaluation.

How It Actually Works

Most tasks you care about are chains. Read the table, compute, round, answer with units. Call the chance of getting one step right pp, per-step accuracy in plain words. Call the number of steps that must all land kk, chain length. Call the chance of getting the whole answer right PP, exact match rate. Under the simplifying view that steps succeed or fail on their own, the whole chain lands only when every step lands, so they multiply: P=pkP = p^k.

That little exponent does all the work. When pp is middling, raising it to the fifth or sixth power crushes PP near zero. When pp gets high, the same curve bends upward fast. The underlying skill moves smoothly. The score you ship on stays flat, then jumps.

Worked example with five steps

Fix the chain at k=5k = 5 steps and walk pp upward:

  • At p=0.5p = 0.5: P=0.55=0.03125P = 0.5^5 = 0.03125, about 0.030.03. The model gets half the steps right on average and almost never the full answer.
  • At p=0.7p = 0.7: square to 0.72=0.490.7^2 = 0.49, square again to 0.74=0.24010.7^4 = 0.2401, multiply once more by 0.70.7 to get 0.168070.16807, about 0.170.17. A big gain per step, still a poor task score.
  • At p=0.9p = 0.9: 0.92=0.810.9^2 = 0.81, 0.93=0.7290.9^3 = 0.729, 0.94=0.65610.9^4 = 0.6561, 0.95=0.590490.9^5 = 0.59049, about 0.590.59. Now most answers land whole.

Notice the asymmetry. Lifting pp from 0.50.5 to 0.70.7 adds 0.200.20 of skill and only 0.140.14 of task score. Lifting it the same 0.200.20 from 0.70.7 to 0.90.9 adds 0.420.42 of task score. Same practice, very different payoff. That is the jump in the diagram: the support curve for pp rises steadily while the focal curve for PP sleeps, then climbs steeply once every step starts clicking.

What this view assumes, and what it costs

The multiplication assumes steps fail independently with the same pp. Real steps share context, so errors correlate and the formula is a lens, not a forecast. Use it to understand the shape, never to predict the scale at which your task switches on.

The trade-off sits in your metrics. Exact match decides what you can ship, but it hides progress: a team watching only PP will think nothing is improving for a long time. Per-step averages reveal progress, but they overstate readiness: 0.70.7 per step sounds decent until you learn the shipped rate is 0.170.17. Track both. When a small model scores zero exact match, check the per-step numbers before you declare the approach dead, and check benchmark contamination before you declare a big jump real. Skills like in-context learning are worth a look here too, since they are the kind of chain-heavy behavior that appears late.

Do not plan a launch around a hoped-for jump. If your feature needs a five-step chain to land and no model you can test today lands it, more compute might help but nothing promises it. Build on demonstrated capability and treat any future jump as upside.

Code

The snippet below is the check behind the worked example: one function, the three per-step values, and the exact match rates they produce.

def exact_match(per_step: float, steps: int) -> float:    """Full task rate when each of `steps` steps lands with `per_step` chance."""    return per_step**steps

for p in (0.5, 0.7, 0.9):    print(p, round(exact_match(p, 5), 4))    # -> 0.5 0.0312    # -> 0.7 0.1681    # -> 0.9 0.5905

The three printed pairs match the hand computation above: 0.030.03, 0.170.17, 0.590.59. Same 0.200.20 climb in pp, small payoff first, large payoff later.

Watch Out For

Reading a flat zero as zero progress

Symptom: your small model scores 00 exact match across sizes, so the team concludes the prompting or training direction is hopeless and drops it.

Fix: log a continuous metric alongside the strict one, such as per-step accuracy or mean token probability on the answer span. If that line rises while exact match sleeps, the direction is working and the chain just is not complete yet. If both lines sleep, then consider a new direction.

Budgeting on a jump that has not happened

Symptom: the roadmap assumes the next scale-up will switch on your specific multi-step workflow, with dates and demos attached to that hope.

Fix: cut the dependency. Ship what current models demonstrate today, keep the hoped-for skill as a tracked experiment with its own eval, and spend only what the continuous metric justifies. A jump you cannot measure in advance is not a milestone.

The Quick Version

  • Emergent abilities are sharp rises in full task success that appear only past a scale threshold.
  • The usual cause is scoring, not magic: per-step skill improves smoothly while exact match demands every step at once.
  • Five steps at 0.70.7 each give about 0.170.17 exact match, so a model can look stuck while it is steadily improving.
  • Track a continuous metric next to exact match, or you will mistake a sleeping score for a dead direction.
  • Never schedule a launch on a capability jump you have not measured yet.