Decoding Strategies

The model hands you odds for the next word. Decoding is how you pick one: always the favourite, a weighted draw, or several guesses at once.

From one next-token distribution, greedy takes the top token, sampling draws from the whole distribution, and beam search tracks two candidates at once

Why Does This Exist?

A trained language model never hands you a word. It hands you odds: at each step, one probability for every token in its vocabulary, tens of thousands of candidates. How LLMs work traces where those odds come from. This page settles the next question, how you turn odds into one token, and why the same odds can produce very different text.

Always taking the favourite feels like the responsible choice. In practice it writes flat copy. The safest word wins each step, then the win makes it likelier next step, so phrases loop and variety dies. If your generations feel dull or stuck, the model may be fine and the picking rule may be at fault.

This page compares the three families everything else refines: greedy, sampling and beam search, plus the top-k and top-p cutoffs that tame sampling's tail. For depth, read temperature sampling for the randomness knob, top-p sampling and top-k sampling for the cutoff rules, beam search for search in full, and constrained decoding when the answer must parse as JSON or match a schema.

Think of It Like This

Friday film night, picked three ways

Your flat votes before every film night. One Friday the slips read Dune 665, Jaws 245, Cats 90 out of a thousand.

You could always show the winner. Simple. By week six, attendance collapses, because the answer never changes.

You could draw one slip from the jar instead. Dune shows most weeks, Jaws gets its nights, Cats shows up rarely. Varied. Occasionally a bad night.

Or for the anniversary, you could book two small screens, test the top two contenders in front of real crowds, then give the main room to the one that lands better. Pricier. Smarter when the night matters.

It stops here: film votes freeze once cast, but every token you pick rewrites the odds for the next one. And model scores are not votes from people. They're learned numbers, and they can be confidently wrong.

How It Actually Works

Every method below reads one object and returns one token. The object is a logit vector $z$ : raw, unnormalised scores, one per candidate token. The return is the chosen token index $y$ . Between them sits the softmax, which turns scores into probabilities $p$ that sum to 1. Keep that pipeline in mind: scores in, odds out, one pick made.

The one distribution every method reads

Write the candidate scores as $z_1, z_2, \ldots, z_V$ , where $V$ is the vocabulary size and $z_i$ is the raw score for token $i$ . With temperature $T$ , a positive knob that defaults to 1, the probability of token $i$ is

p_i = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}

Here $\exp$ is the exponential function and the sum runs over every candidate token $j$ , so the $p_i$ values sum to 1. Lower $T$ sharpens the odds toward the leader. Higher $T$ flattens them toward uniform. $T = 1$ leaves the model's scores as they are.

Greedy: take the top token

Greedy decoding picks $y = \arg\max_i p_i$ , the index $i$ with the largest probability, every step, with no dice involved. Same prompt, same output, always. It costs one pass and nothing else, which is why it rules factual lookups and short answers.

Don't ask it to entertain you. For stories, brainstorming or chat, its love of the safe word reads as bland, and its loops read as broken. When the brief rewards variety, greedy is the wrong default.

Sampling with temperature: draw, then reshape

Sampling draws $y \sim p$ , meaning each token is picked with its own probability: a token at 30 percent comes up about 30 percent of the time. Rerun the prompt and you'll get a different answer, which is the point. Temperature reshapes before the draw: divide the logits by $T$ first, then softmax. Values below 1 bunch mass on the leader. Values above 1 spread it around. The full treatment, with the sharpening math, is in temperature sampling.

Use it when variety beats repeatability. Don't use it where the answer must be stable, such as graded extraction or tool calls with strict schemas. Randomness there is a bug, not a feature.

Beam search: keep B paths alive

Beam search refuses the single-path gamble. It keeps $B$ live candidates, where $B$ is the beam width you choose, extends each by its likeliest next tokens, scores every extension by its summed log probabilities, and carries only the top $B$ forward. A weak start can still win if its continuation is strong. The complete walkthrough is in beam search.

You pay for the insurance. Work and memory grow roughly with $B$ , and you'll feel it in latency too. For open chat the winner is often the most generic sentence in the room. Reach for beams when one correct answer exists, as in translation or transcription. Leave them home for open-ended writing.

Top-k and top-p: cut the tail, then draw

Plain sampling can pull a one-in-a-thousand weirdo. The cutoffs stop that. Top-k sampling keeps the $k$ most probable tokens, where $k$ is a count you set, zeroes the rest, then renormalises, which means dividing the survivors by their sum so they total 1 again. Top-p sampling, also called nucleus sampling, keeps the smallest set whose probabilities add up to at least $P$ , where $P$ is a mass target like 0.9, then renormalises the same way. Small $k$ or $P$ means safe and narrow. Large means wild and wide. Details and edge cases are in top-k sampling and top-p sampling. When the shape of the answer is fixed, masking goes further: constrained decoding zeroes every token the schema forbids before anyone draws.

The whole page rests on one assumption: higher probability means more desirable. That holds when the model is well calibrated and the task rewards the likely answer. It breaks when safe phrasing outscores good phrasing, or when each step's local winner leads the full sentence astray. Greedy and wide beams suffer that myopia most.

Worked example: logits [2, 1, 0]

Call the three candidates cat, dog and bird, with logits $z = [2, 1, 0]$ . At $T = 1$ , exponentiate: $[\exp(2), \exp(1), \exp(0)] = [7.389, 2.718, 1.000]$ . Sum: $11.107$ . Divide: $p = [0.665, 0.245, 0.090]$ . Greedy takes cat. Sampling draws cat about two thirds of the time, dog about a quarter, bird about one time in eleven. Beam search with $B = 2$ keeps cat and dog as its two live tracks.

Turn the knob to $T = 2$ . Scores become $[1.0, 0.5, 0.0]$ , exponentials $[2.718, 1.649, 1.000]$ , sum $5.367$ , so $q = [0.506, 0.307, 0.186]$ . Same ranking, softer gaps. Bird jumps from 9 percent to nearly 19 percent, which previews the first pitfall below.

Apply nucleus cutoff $P = 0.9$ to the $T = 1$ odds. Sorted mass accumulates $[0.665, 0.910, 1.000]$ , so the smallest set reaching 0.9 is cat plus dog. Renormalise by their sum $0.910$ : $[0.665 / 0.910, 0.245 / 0.910] = [0.731, 0.269]$ . Bird's chance falls to zero. Those are the bars and boxes in the diagram, and the numbers the snippet reprints.

Code

The snippet reprints the worked example, so you can check the arithmetic instead of trusting it.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max())  # shift so exp sees 0 as its largest input6    return e / e.sum()7
8
9z = np.array([2.0, 1.0, 0.0])  # logits for cat, dog, bird10p = softmax(z)11print([f"{v:.3f}" for v in p])  # -> ['0.665', '0.245', '0.090']12print(int(np.argmax(p)))  # -> 013q = softmax(z / 2.0)  # temperature T = 2.0 flattens the same scores14print([f"{v:.3f}" for v in q])  # -> ['0.506', '0.307', '0.186']15r = p[:2] / p[:2].sum()  # top-p 0.9 keeps cat and dog, mass 0.91016print([f"{v:.3f}" for v in r])  # -> ['0.731', '0.269']

Greedy prints 0, the index of cat. The first list matches the $T = 1$ odds, the second the flattened $T = 2$ odds, the third the nucleus pair. If your library's softmax disagrees past the third decimal, suspect your rounding, not the math.

Watch Out For

Sampling the full tail lets one strange token wreck a good answer

Most draws look fine, then bird appears in a sentence about cats and dogs and the whole answer tilts. The symptom is rare, fluent nonsense: nine good generations, one bizarre word that slipped through at low probability. At $T = 2$ in the example above, bird nearly doubles from 0.090 to 0.186, so roughly one draw in five goes weird.

Cap the tail before you draw. Set top-p to 0.9 or top-k to a small count so the hopeless candidates hold zero mass, and keep temperature moderate for anything factual. Save high temperatures for brainstorming, where a strange word costs nothing.

Passing temperature 0 instead of calling greedy

Zero sounds like the deterministic setting, but the formula divides by $T$ , so $z / 0$ is undefined, not clever. The symptom is an API error, a silent clamp to some tiny value, or code that works on one provider and throws on the next.

Branch instead. When you want the top token every time, call the argmax path directly and validate that $T$ stays above 0 wherever sampling runs. Determinism deserves its own code path, not an edge case of a division.

The Quick Version

Decoding turns the model's next-token odds into one token, and the same odds give different text under different rules.
Greedy decoding always takes the top token, so it is cheap and repeatable but dull and loopy on open-ended writing.
Temperature sampling draws by probability, with values below 1 sharpening the favourite and values above 1 spreading mass to the tail.
Beam search carries several live candidates and scores each by summed log probability, which suits translation but drags on chat.
Top-k and top-p cut the unlikely tail before the draw, and top-p 0.9 on odds of 0.665, 0.245 and 0.090 keeps two tokens at 0.731 and 0.269.