Skip to content
AI360XpertBeta

Test-Time Compute in LLMs

Spend extra effort at answer time, not training time. You sample several tries and keep the one that checks out.

Sampling five answers and keeping the highest scored one lifts success coverage from 0.30 to about 0.83
Sampling five answers and keeping the highest scored one lifts success coverage from 0.30 to about 0.83

Why Does This Exist?

One greedy answer fails you on hard questions. You ask for a proof, a tricky bug fix, or a multi-step plan, and the model's first draft looks smooth while hiding a wrong turn in step two. You could train a bigger model, but that bill grows fast and needs far more data. So you want a second lever, one you can pull per question.

Test-time compute is that lever. You keep the weights frozen and spend extra work while answering: sample several tries, score them, and return the one that checks out. Same model, better odds.

This page covers that inference-side trade: how sampling plus a checker lifts coverage, what it costs, and when it flops. The single written trace it scales is chain of thought. The vote-based cousin is self-consistency. The training-side budget question lives in compute-optimal training.

You already need one idea. Sampling means drawing the next token from a distribution instead of always taking the top one, so the same prompt can produce different answers. Temperature controls how spread out that distribution is, and the full menu of sampling rules is in decoding strategies.

Think of It Like This

Extra time and scrap paper in an exam

You sit a hard math exam two ways. First way: you studied for a year, then you must answer in thirty seconds flat with no rough paper. Second way: you studied as usual, but you get thirty extra minutes and five sheets of scrap paper. You try five approaches, cross out four, and hand in the clean one.

The five rough sheets are your sampled traces. Crossing out is the verifier. Handing in one sheet is selection. That's the whole trick, and it's why extra time helps most on questions where checking an answer is easier than finding it.

Where it stops: scrap paper can't teach you what you never learned. If you cannot recognise the right answer when you see it, five tries just give you five confident wrong sheets. Extra tries amplify a decent solver with a decent checker. They do not replace either one.

How It Actually Works

The common setup is sample-then-pick, also called best-of-NN. You run the same prompt NN times with sampling turned on, score each answer, and keep the top score. Let NN be the number of tries, let pp be the chance one try succeeds on its own, let sis_i be the verifier score for try ii, and let y⋆y^{\star} be the picked answer, the try with the largest sis_i. In words: NN sets your bill, pp sets your starting odds, sis_i is each try's grade, and y⋆y^{\star} is the winner you actually show.

The parallel trick: sample, score, keep one

First you diversify. With temperature above zero the model walks different paths: different first steps, different algebra, different plans. Then a checker grades each path. That checker can be a second model, a rule like unit tests, or a person. You return the argmax, the try with the best grade. If the checker is solid, your coverage climbs even though the base model never changed.

There's a sequential flavor too: let one trace run longer, with backtracking and self-checks. That's still test-time compute, just spent on depth instead of breadth. This page follows the parallel version because its maths is cleaner and its diagram reads at a glance.

A worked example with five tries

Say one try solves your problem with chance p=0.30p = 0.30. You take N=5N = 5 independent tries. Write q=1−pq = 1 - p for the chance one try fails, so q=0.70q = 0.70. All five fail with chance qNq^N, and at least one succeeds with chance 1−qN1 - q^N. That quantity is called pass@k, with k=Nk = N.

Now compute it. q2=0.70×0.70=0.49q^2 = 0.70 \times 0.70 = 0.49. Square again for q4=0.49×0.49=0.2401q^4 = 0.49 \times 0.49 = 0.2401. Multiply once more for q5=0.2401×0.70=0.16807q^5 = 0.2401 \times 0.70 = 0.16807. So coverage is 1−0.16807=0.831931 - 0.16807 = 0.83193, about 0.830.83. You moved from 0.300.30 to about 0.830.83 without touching training.

Scoring picks the winner the same way. Suppose your five tries earn verifier scores s=[0.15,0.31,0.82,0.47,0.66]s = [0.15, 0.31, 0.82, 0.47, 0.66]. The max is 0.820.82 at index 22, the middle try, so y⋆y^{\star} is try 33. The diagram above draws exactly this: five boxes, those five scores, and the middle one kept. Tokens tell the price. At 120120 tokens per try, five tries cost 5×120=6005 \times 120 = 600 tokens against 120120 for one. Five times the bill for roughly that 0.300.30 to 0.830.83 lift, assuming the verifier actually finds the good one.

That last clause is the assumption holding the whole thing up. The 0.830.83 is coverage, the chance a good answer exists somewhere in the five. What you ship is only as good as your pick. A perfect checker ships 0.830.83. A weak one ships whatever sounds fluent.

What it costs and when to skip it

The trade is linear cost for shrinking gains. Each extra try adds about the same token bill while adding less coverage than the last, because 1−qN1 - q^N flattens as NN grows. You also pay latency unless you run tries side by side, and side by side still costs you money.

Don't use it for easy recall, strict latency, or tight budgets. If one greedy answer already works, if users wait on every token, or if you've got no checker better than vibes, extra samples burn cash for nothing. Use it where answers are checkable: code with tests, maths with a final answer, plans with hard constraints. Pair it with beam search when you want search inside one decode, and with an LLM judge when a model does the grading.

Code

The point here is the arithmetic, not an API. This snippet computes the coverage from the worked example and picks the winner by score.

def pass_at_k(p: float, k: int) -> float:    """Chance at least one of k tries succeeds, given per-try chance p."""    return 1.0 - (1.0 - p) ** k

def pick_best(scores: list[float]) -> int:    """Index of the highest verifier score."""    return max(range(len(scores)), key=lambda i: scores[i])

scores: list[float] = [0.15, 0.31, 0.82, 0.47, 0.66]print(round(pass_at_k(0.3, 5), 5))  # -> 0.83193print(pick_best(scores))  # -> 2print(max(scores))  # -> 0.82

Coverage is 0.831930.83193, the winner sits at index 22, and its score is 0.820.82. Those are the same three numbers the prose and the diagram carry.

Watch Out For

A weak checker picks a fluent wrong answer

Extra samples only help if the pick is better than chance. A weak verifier grades style instead of substance, so as NN grows it gets more chances to fall for a confident fake. The symptom is grim: coverage goes up while shipped accuracy stalls or drops, and the chosen answers read beautifully.

Fix it before you raise NN. Score a small labelled set by hand, check the verifier agrees with you, and watch picked accuracy as you scale from one to five to ten tries. If the curve flattens early, fix the checker or switch checkers, for example tests for code or a stricter judge, before buying more samples.

Five tries that are secretly one try

If temperature is zero, or every try reuses the same seed and prompt, your five samples come back byte-identical. You paid five times for one answer. The symptom is easy to spot: identical outputs, or near-identical ones that differ by a comma.

Fix the diversity first. Raise temperature a notch, vary the prompt wording or few-shot examples across tries, and eyeball a few batches side by side. Do not raise N until the tries actually differ. No spread means no search, whatever NN says.

The Quick Version

  • Test-time compute spends budget while answering, sampling several tries and shipping the best-checked one.
  • With per-try success at 0.3, five independent tries lift coverage to about 0.83 under a perfect checker.
  • The shipped answer equals the checker's pick, so a weak checker caps the whole trick no matter how large N grows.
  • Cost grows about linearly in tries while coverage gains shrink, so this fits checkable work like code and maths, not casual chat.
  • If the tries look identical, fix sampling diversity before paying for more of them.