Top-K Sampling
Keep only the K most likely next tokens and throw the rest away, then draw from the survivors. You keep some surprise without ever drawing a hopeless outsider.
Why Does This Exist?
Your model never hands you a word. It hands you odds, one score for every token in a vocabulary of tens of thousands. Next-token prediction is where those odds come from, and decoding strategies is the menu of ways to turn them into one pick.
Plain sampling draws straight from the full odds. Most draws land near the top, but the tail holds thousands of hopeless candidates with tiny, nonzero chances. Generate a long answer and you have rolled that die once per token, so a one in a thousand weirdo stops being rare. One bad draw tilts the sentence, and every step after it builds on the tilted version.
Top-k sampling is the simplest guard against that accident. Keep the most likely tokens, set everything else to zero, then draw from the survivors. You still get variety, but the hopeless tail can never come up. This page covers that fixed count rule and its arithmetic. The knob that reshapes the odds before the cut lives in temperature sampling, and the cousin that sizes the kept set from the odds themselves lives in top-p sampling.
The one idea you need first: raw scores are logits, plain numbers that can be negative and sum to nothing in particular. Softmax turns them into probabilities that sum to 1: exponentiate each score, add them up, divide each by the total. Once you have those probabilities, sampling means drawing each token with its own chance. Top-k sits between those two steps: after the odds exist, before the draw happens.
Think of It Like This
A hiring shortlist with a fixed interview count
You post one role and 1,000 people apply. You rank every application from strongest to weakest.
Plain sampling would interview by weighted lottery over all 1,000, so the 999th application still holds a ticket. Most days you would meet someone good. Some unlucky day you would spend an hour with someone you should never have called.
So you set a rule: keep the top 6, shred the other 994, then run the weighted lottery among the 6 survivors. You have kept the draw, but you have deleted disaster. The number 6 is . A bigger means a wider, wilder shortlist. means no lottery at all, just the favourite every time.
It stops here: resumes sit still once ranked, but token odds rewrite themselves after every pick you make. And a weak resume wastes one interview, while a bad token poisons the context the next scores are built from, so one slip compounds faster than one bad hire.
How It Actually Works
Write the vocabulary size as , the count of candidate tokens. Write the raw score for token as , its logit. Write its probability after softmax as , so the values sum to 1 over all candidates. Write your chosen count as , a whole number from 1 up to . Write the odds after the cut as .
The cut runs the same four moves every step:
- Sort the values from largest to smallest.
- Keep the first . Call their total kept mass , the sum of those probabilities.
- Set every other to 0. The tail is gone, not merely unlikely.
- Divide each survivor by , so for the kept tokens. Now the survivors sum to 1 and you draw one token from .
Ranking never changes here. The leader before the cut still leads after it, just with a larger share, because every survivor is divided by the same , which sits below 1.
Worked example: K = 3 on six odds
Say the sorted odds are . They sum to 1.00, so they are valid probabilities. Take .
Kept mass is . Removed mass is , and those three tokens go to exactly 0.
Divide the survivors by :
Rounded to three places that is , which sums to 1.000. The leader jumps from 0.40 to half the draw, not because it got better, but because one fifth of the mass was deleted and handed back to the survivors. Those three bars and the cut between ranks 3 and 4 are the picture above, and the snippet below reprints the same division.
The trade: safety for rigidity
You buy safety with rigidity, and the price shows at both ends. When the model is confident, a fixed keeps junk to fill its quota. If the top two tokens already hold 0.95 of the mass and you still keep 40, you have re-invited 38 losers you meant to ban. When the model is flat, the same cuts gold. If 100 decent continuations split the mass and you keep 10, you have thrown away 90 live options and narrowed text that should have stayed wide.
So do not use a small fixed where variety is the point, and do not use a large fixed where stability is the point. Graded extraction, math with a checked answer, and tool calls with strict shapes want the top token or constrained decoding, which forbids illegal tokens outright, not a lottery among . Save top-k for open writing where a little surprise helps, pair it with temperature as temperature sampling shows, and reach for top-p sampling when the shape of the odds changes a lot between steps.
The rule rests on one bet: higher probability means more desirable, and the ranking itself is worth trusting. That holds when the model is well calibrated and the task rewards the likely answer. It fails when safe phrasing outscores good phrasing, or when the scores are wrong outright. A cutoff cannot fix bad scores. It only decides which bad scores are allowed to win.
Code
The snippet replays the worked example, so you can check the division instead of trusting it.
import numpy as np
def top_k_renormalise(probs: np.ndarray, k: int) -> np.ndarray: keep = np.argsort(probs)[::-1][:k] # indices of the K largest odds q = np.zeros_like(probs, dtype=float) q[keep] = probs[keep] # copy survivors, tail stays exactly 0 return q / q.sum() # divide by kept mass S so survivors sum to 1
p = np.array([0.40, 0.25, 0.15, 0.10, 0.06, 0.04]) # sorted odds from aboveq = top_k_renormalise(p, 3)print([f"{v:.3f}" for v in q]) # -> ['0.500', '0.312', '0.188', '0.000', '0.000', '0.000']print(f"{p[:3].sum():.2f}") # -> 0.80The two prints match the hand division: kept mass 0.80, survivors at 0.500, 0.312 and 0.188, tail at zero. If your library disagrees past the third decimal, suspect rounding, not the math.
Watch Out For
Stacking top-k on top of top-p until P does nothing
You set and together, then wonder why moving changes nothing. The symptom is a dead knob: and return the same variety.
Most stacks run top-k first, so already shrinks the pool before top-p ever sees it. If those 10 hold less than , the second cutoff keeps everything it receives.
Pick one cutter. Leave very large or off and tune alone, as top-p sampling shows, or set a deliberate and leave at 1.0. Do not tune both at once unless you can say which one bit.
Setting K = 1 when you meant safe sampling
You want safe but lively text, so you push down to 1. The symptom is deja vu: reruns return the same sentence, stories loop, and temperature stops mattering.
keeps one token and zeroes the rest, so the draw has one outcome. That is greedy decoding with extra steps, and no temperature or seed restores variety because there is nothing left to vary.
Keep at a small handful for lively work, and call greedy by name when you want determinism, the argmax rule in decoding strategies. If reruns match exactly and you did not ask for that, check before you blame the model.
The Quick Version
- Top-k sampling keeps the most likely tokens, zeroes the rest, then draws from the survivors.
- With odds of 0.40, 0.25, 0.15, 0.10, 0.06 and 0.04, keeps mass 0.80 and renormalises the survivors to 0.500, 0.312 and 0.188.
- A fixed is rigid: it keeps junk to fill its quota when the model is confident and cuts live options when the model is flat.
- is greedy decoding with extra steps, while a very large behaves like plain sampling from the full tail.
- Pair sampling with one cutoff at a time, and use greedy or constrained decoding where the answer must stay stable.