Top-P (Nucleus) Sampling
Top-p keeps the smallest set of next tokens whose odds add up to your target, so a confident model keeps one or two options and an unsure model keeps many.
Why Does This Exist?
Your model never hands you a word. It hands you odds, one chance per token across tens of thousands of candidates. Next-token prediction is where those odds come from, and decoding strategies is the menu for turning them into one pick.
Plain sampling draws from the full odds. Most draws land near the top, but the tail holds thousands of hopeless candidates with tiny nonzero chances. Write a long answer and you roll that die once per token, so a one-in-a-thousand weirdo stops being rare. One bad draw tilts the sentence, and every step after it builds on the tilted version.
Top-p sampling is the guard that adapts. You may also see it called the nucleus, the smallest head of the list that carries your target mass. Instead of keeping a fixed count, you keep adding tokens from the top until their total hits your target, then you draw from the survivors. Confident odds collapse to one or two keepers. Flat odds keep many.
This page covers that mass rule and its arithmetic. The fixed-count cousin lives in top-k sampling. The knob that reshapes odds before either cut lives in temperature sampling. The relative-threshold alternative lives in min-p sampling.
One idea first: raw scores are logits, plain numbers that can be negative and sum to nothing in particular. Softmax turns them into probabilities that sum to 1. You exponentiate each score, add everything up, divide each by the total. Top-p sits after those odds exist and before you draw.
Think of It Like This
Packing to a weight limit, not a count
You fly with a 50-pound bag rule. You lay gifts on the bed from most wanted to least, then load in that order until the scale reads 50. Heavy gifts fill the bag fast. You might stop after three. Light gifts barely move the needle. You might load thirty before you hit the mark.
Point for point: gifts in order are tokens sorted by probability, weight is probability mass, 50 pounds is your target , and stopping is the cutoff. The draw among packed gifts is sampling from survivors.
It stops here: gifts keep their weight, but token odds rewrite themselves after every pick you make. And repacking a bag costs you minutes, while a bad token poisons the context the next scores are built from, so one slip compounds faster than one heavy souvenir.
How It Actually Works
Write the vocabulary size as , the count of candidate tokens. Write the raw score for token as , its logit. Write its probability after softmax as , its share of the total, so all values sum to 1. Write your target as , a fraction such as 0.9. Write the total of the kept tokens as , the kept mass. Write the final odds as , the renormalised shares you actually draw from.
The cut runs the same moves each step:
- Sort the values largest first.
- Walk down the order, adding to a running total.
- Stop at the first token where the total reaches or passes . Keep every token up to and including that one.
- Set the rest to exactly 0. The tail is gone, not merely unlikely.
- Divide each survivor by , so for kept tokens. Now survivors sum to 1 and you draw once from .
Ranking never changes. The leader before the cut still leads after it, just with a larger share, because every survivor divides by the same below 1. Temperature still matters here: it reshapes before this rule ever sees them, as temperature sampling shows.
The cutoff in plain sums
You do not need the full vocabulary to see it. Take three sorted odds. Add the first. Below target, continue. Add the second. At or above target, stop and cut. That is the whole test, and the picture above draws it twice on one shared height scale so sharp and flat stay honest.
Worked example: sharp odds keep 2
Say the sorted odds are . They sum to 1.000. Take .
Running totals: 0.665, then , then 1.000. The first total sits below 0.9. The second reaches 0.910, so you stop. Keep the first two, cut the third.
Kept mass is . Removed mass is 0.090, gone to zero. Divide:
Rounded that is , summing to 1.000. The leader jumps from 0.665 to 0.731, not because it improved, but because the deleted 0.090 was handed back to survivors. Those two solid bars and the dashed cut bar are the left panel above, and the snippet below reprints the same division.
Worked example: flat odds keep 5
Say the sorted odds are . They sum to 1.00. Take the same .
Running totals: 0.25, 0.47, 0.65, 0.80, , then 1.00. You pass 0.9 only at the fifth token, so you keep five and cut one.
Kept mass is . Removed mass is 0.08. Divide each survivor by 0.92:
That is , summing to 1.000. Same rule, same , different kept count. That adaptivity is the right panel above, and it is why top-p fits steps where the shape swings a lot.
The trade and when to skip it
You buy adaptivity with trust in the ranking. When odds are sharp you get safety without a quota to tune. When odds are flat you keep variety without hand-picking a count. The cost is a new knob plus sensitivity to bad scores. A cutoff cannot fix wrong odds. It only decides which wrong odds may win.
So skip sampling where the answer must stay stable. Graded extraction, math with a checked answer, and tool calls with strict shapes want the top token or constrained decoding, which forbids illegal tokens outright. Save top-p for open writing, pair it with a moderate temperature, and do not stack two cutters at once.
The rule rests on one bet: higher probability means more desirable. It holds when the model is well calibrated and the task rewards the likely answer. It fails when safe phrasing outscores good phrasing, or when a crowd of weak tokens pools mass to pass together. Sixty tokens at 0.01 each carry 0.60 together while none deserves a draw. That pooling failure is what min-p sampling was built to refuse.
Code
The snippet replays both worked examples, so you can check the divisions instead of trusting them.
import numpy as np
def top_p_renormalise(probs: np.ndarray, p: float) -> np.ndarray: order = np.argsort(probs)[::-1] # largest first cum = np.cumsum(probs[order]) # running total in sorted order k = int(np.searchsorted(cum, p, side="left")) + 1 # first index reaching P keep = order[:k] # smallest set hitting P q = np.zeros_like(probs, dtype=float) q[keep] = probs[keep] # tail stays exactly 0 return q / q.sum() # divide by S so survivors sum to 1
sharp = np.array([0.665, 0.245, 0.090]) # sorted odds, sum 1.000q_sharp = top_p_renormalise(sharp, 0.9)print([f"{v:.3f}" for v in q_sharp]) # -> ['0.731', '0.269', '0.000']print(f"{sharp[:2].sum():.3f}") # -> 0.910flat = np.array([0.25, 0.22, 0.18, 0.15, 0.12, 0.08]) # sum 1.00q_flat = top_p_renormalise(flat, 0.9)print([f"{v:.3f}" for v in q_flat]) # -> ['0.272', '0.239', '0.196', '0.163', '0.130', '0.000']print(f"{flat[:5].sum():.2f}") # -> 0.92The prints match the hand arithmetic: kept mass 0.910 with survivors at 0.731 and 0.269, then kept mass 0.92 with five survivors. If your library keeps one extra token past the crossing point, you are reading an inclusive-versus-exclusive variant. Name the variant before you compare numbers.
Watch Out For
Setting P so low the draw turns greedy
You want tidy output, so you drop to 0.5. The symptom is deja vu: reruns return the same sentence, stories loop, and temperature stops mattering.
With the running total often passes on the first token, so the kept set holds one survivor and the draw has one outcome. That is greedy decoding with extra steps.
Keep around 0.85 to 0.95 for open writing. Call greedy by name when you want determinism, the argmax rule in decoding strategies. If reruns match exactly and you did not ask for that, check before you blame the model.
Stacking top-k on top-p until P does nothing
You set a small and together, then wonder why moving changes nothing. The symptom is a dead knob: and return the same variety.
Most stacks run top-k first, so a tight already shrinks the pool before top-p sees it. If those survivors hold less than , the second cutoff keeps everything it receives.
Pick one cutter. Leave very large or off and tune alone, or set a deliberate as top-k sampling shows and leave at 1.0. Do not tune both at once unless you can say which one bit.
The Quick Version
- Top-p sampling keeps the smallest head of the sorted odds whose total reaches , zeroes the tail, then divides survivors by kept mass to sum to 1.
- With odds of 0.665, 0.245 and 0.090, keeps two tokens at mass 0.910 and renormalises them to 0.731 and 0.269.
- With odds of 0.25, 0.22, 0.18, 0.15, 0.12 and 0.08, the same keeps five tokens at mass 0.92 and renormalises them to 0.272, 0.239, 0.196, 0.163 and 0.130.
- A fixed count keeps junk to fill its quota when odds are sharp and cuts live options when odds are flat, while top-p sizes the kept set from the odds themselves.
- Skip top-p where answers must stay stable, such as graded extraction or strict tool calls, and use greedy or constrained decoding there instead.
- The rule assumes higher probability means more desirable, so on miscalibrated scores or pooled weak mass it keeps the wrong tokens with confidence.