Skip to content
AI360XpertBeta

Speculative Decoding

A tiny model drafts the next few words and the big model checks them all at once. You pay for one big pass but can keep several tokens.

A small draft model proposes three tokens in a row and the large target model checks all three in one parallel pass, keeping two and replacing the third
A small draft model proposes three tokens in a row and the large target model checks all three in one parallel pass, keeping two and replacing the third

Why Does This Exist?

You generate text one token at a time. Each step runs the big model once, and the expensive part is not the arithmetic but moving its weights from memory to the chips, again, for every single token. That's why long answers feel slow even on strong hardware: the cost is per pass, not per idea.

Here is the mismatch this page exploits. Producing one token costs a full pass, but checking several tokens costs that same single pass, because the model scores every position at once. If someone hands you three good guesses, you can grade all three for the price of generating one. Next-token prediction sets up why generation runs in sequence, and decoding strategies covers how each token gets picked. This page covers the speedup built on top: guess cheaply, verify in bulk.

Its neighbours answer the adjacent questions. The KV cache stops you from recomputing past tokens, paged attention stores that cache without waste, and prefix caching shares it across requests. None of them reduces the number of big passes per answer. That's this page's job.

Think of It Like This

A prep cook and a head chef at the pass

A head chef plates every dish herself, one at a time, and the line crawls. So she stations a quick prep cook ahead of her: he plates the next three dishes as his best guess at her style and slides them down the pass. She tastes all three in one go. Two are right, so they go out. The third needs salt, so she fixes that plate herself and bins the rest of his guesses.

The mapping holds plate for plate. The prep cook is the draft model: fast, cheap, and wrong sometimes. Tasting is verification: far cheaper than cooking from scratch. A plate going out is an accepted token. The fixed salty plate is the replacement sample, and everything after the first miss gets remade because it was seasoned on top of a mistake.

It stops here: a chef's verdict is taste, and a confident cook can argue with it. The target model's verdict is the definition of correct output, so there is nothing to argue, and the guarantee below is exact rather than a matter of opinion.

How It Actually Works

Two models share the work. The draft model is small and quick: call its next-token distribution q(x)q(x), meaning the probability it assigns to token xx. The target model is the large one whose output you actually want: call its distribution p(x)p(x). Both read the same prefix. Only the target's taste counts.

Draft a few, verify them together

The draft model generates γ\gamma tokens one after another, where γ\gamma (gamma) is a small count you choose, such as 3. Because the draft model is tiny, those γ\gamma steps cost little. Then the target model runs exactly one forward pass over the prefix plus all γ\gamma drafts. In that single pass it produces its own distribution at every draft position, plus one extra distribution for whatever should come next. Scoring positions in parallel is the whole trick: one big pass grades the entire batch.

The acceptance rule

Walk the drafts left to right. For draft token xix_i at position ii, keep it with probability min⁡(1,pi(xi)/qi(xi))\min(1, p_i(x_i) / q_i(x_i)), the smaller of 1 and the ratio of target probability to draft probability. In practice you draw rr, a uniform random number between 0 and 1, and keep the token when rr falls below that ratio. When the target likes the token at least as much as the draft did, the ratio is 1 or more, so you always keep it. When the target likes it less, you keep it only sometimes, in exact proportion to how much less.

The first miss ends the round. Throw away that draft and every draft after it, then sample one replacement token from the leftover mass, which is the target distribution minus the draft distribution, floored at zero and renormalised. If every draft is kept, sample one extra bonus token from the target's final distribution instead. Either way the round banks between 1 and γ+1\gamma + 1 tokens for a single target pass, and the mathematics leaves the output distribution identical to sampling from the target alone. That lossless property is the point: same text, fewer big passes.

This rests on one assumption: the draft distribution sits close to the target's. Call the average keep chance the acceptance rate. High acceptance means most rounds bank several tokens and you win. Low acceptance means most rounds bank a single token while still paying for the drafts, and you lose. So don't reach for this when the draft and target disagree often, which is what the pitfalls below spell out.

Worked example: three drafts, two keeps and one fix

The prefix is "The cat". The draft model proposes γ=3\gamma = 3 tokens: "sat", "on", "mat". The two models assign these probabilities to the drafted tokens:

  • "sat": q1=0.70q_1 = 0.70, p1=0.80p_1 = 0.80. Ratio 0.80/0.70≈1.140.80 / 0.70 \approx 1.14, capped at 1. Always keep.
  • "on": q2=0.60q_2 = 0.60, p2=0.45p_2 = 0.45. Ratio 0.45/0.60=0.750.45 / 0.60 = 0.75. Draw r2=0.31r_2 = 0.31. Since 0.31<0.750.31 < 0.75, keep.
  • "mat": q3=0.50q_3 = 0.50, p3=0.10p_3 = 0.10. Ratio 0.10/0.50=0.200.10 / 0.50 = 0.20. Draw r3=0.62r_3 = 0.62. Since 0.62>0.200.62 > 0.20, reject.

The round keeps "sat" and "on", discards "mat", and samples a replacement, say "rug", from the target's leftover mass. Final output of the round: "sat", "on", "rug". Three tokens banked for one target pass, where plain decoding would have spent three. Those are the boxes and verdicts in the diagram, and the numbers the snippet replays.

Code

This snippet replays the worked example: three drafts, their target and draft odds, one draw each, and the replace-on-first-miss rule.

def verify(drafts: list[str], p: list[float], q: list[float], draws: list[float]) -> list[str]:    kept: list[str] = []    for tok, pt, qt, r in zip(drafts, p, q, draws):        ratio = min(1.0, pt / qt)  # keep chance for this draft token        if r < ratio:            kept.append(tok)        else:            kept.append("rug")  # first miss is replaced, later drafts discarded            break    return kept

drafts = ["sat", "on", "mat"]p = [0.80, 0.45, 0.10]  # target odds for the drafted tokensq = [0.70, 0.60, 0.50]  # draft odds for the same tokensdraws = [0.90, 0.31, 0.62]  # one uniform draw per position
print(verify(drafts, p, q, draws))  # -> ['sat', 'on', 'rug']

Position one always keeps because the ratio caps at 1 and 0.90<10.90 < 1. Position two keeps because 0.31<0.750.31 < 0.75. Position three misses because 0.62>0.200.62 > 0.20, so "mat" becomes "rug" and the print matches the hand computation above.

Watch Out For

A draft model that rarely agrees with its target

The symptom is a speedup that never arrives: acceptance sits near zero, nearly every round banks a single token, and the system runs slower than plain decoding because you pay for the drafts on top. People usually blame the serving setup. The setup is fine; the draft is just guessing a different style, often a cheap model from another family or one trained on very different text.

Fix it by pairing models that speak alike, ideally a small model from the same family or distilled from the target, and by measuring the acceptance rate before celebrating. If it stays low, shorten γ\gamma to 2 or switch speculation off for that workload. Speculation multiplies agreement; it cannot create it.

Speculating through wide open sampling

The symptom is lopsided latency: crisp, deterministic requests fly while high-temperature creative writing crawls at baseline. At high temperature the target spreads its mass across many candidates, so any single drafted token is unlikely to match, and the acceptance rule rejects most guesses. The drafts were reasonable; the task was just too open to predict well.

Fix it by aiming speculation at low-entropy work: code completion, extraction, short factual answers, anything close to greedy. Save high-temperature brainstorming for plain decoding, or drop γ\gamma down and confirm the acceptance rate recovers before keeping it on.

The Quick Version

  • Speculative decoding drafts several tokens with a small model and verifies them in one target pass, banking between 1 and gamma plus 1 tokens per pass.
  • Each draft is kept with probability min of 1 and target odds over draft odds, and the first miss is replaced from the target's leftover mass.
  • The output distribution matches the large model alone, so quality is unchanged and only the number of big passes drops.
  • Speculative decoding wins when the draft agrees with the target often and loses when it does not, so measure the acceptance rate before trusting it.
  • Skip speculation for wide open high-temperature sampling and mismatched draft models, where most rounds bank a single token while still paying for drafts.