Transformer Decoding

Generating text one token at a time requires an LLM to make a continuous stream of decisions. But an LLM doesn't just output a word—it outputs a probability distribution across its entire vocabulary. How we sample from that distribution drastically changes the tone, creativity, and coherence of the text.

The Raw Logits

At every step, the final layer of the neural network produces logits: raw, unnormalized scores for every possible next token. To convert these scores into a valid probability distribution (where all values sum to 1), we pass them through a softmax function.

The Lottery Ticket

Think of logits as lottery tickets. A token with a high logit has many tickets, giving it a high chance of being drawn. A token with a low logit has very few. But even a low-probability token could be drawn, unless we explicitly rip up its tickets.

Temperature Scaling

Temperature adjusts the confidence of the model by scaling the logits before the softmax function.

• T = 1.0: The default state. Logits are unchanged.
• T < 1.0 (Cooler): Divides logits by a small number, making the differences between them larger. The most likely tokens become even more dominant, and the long tail of unlikely tokens shrinks to near-zero. At T = 0.1, the model becomes highly deterministic and repetitive.
• T > 1.0 (Warmer): Divides logits by a large number, compressing them closer together. This flattens the distribution, giving unlikely tokens a better chance. If pushed too high (e.g., T = 2.0), the model samples almost randomly, generating gibberish.

Truncating the Tail: Top-K and Top-P

Even with temperature scaling, there's always a tiny chance the model picks a completely nonsensical token from the very end of the long tail. To prevent this, we use truncation strategies to explicitly zero out the probabilities of bad options.

Top-K is a hard cutoff. If K = 50, we only keep the 50 tokens with the highest probabilities and zero out the rest. We then re-normalize those 50 so they sum to 1. This prevents the model from ever wandering into the deep tail, but it's rigid—if there are 100 equally good choices, it blindly cuts off half of them.

Top-P (Nucleus Sampling) is a dynamic cutoff. Instead of keeping a fixed number of tokens, it keeps tokens until their cumulative probability hits P. If P = 0.9, it keeps adding the most likely tokens until they account for 90% of the probability mass.

• If the model is very confident, the top 2 tokens might sum to 90%, so it only keeps those 2.
• If the model is unsure, it might take 100 tokens to reach 90%, so it keeps 100.

Top-P adapts to the shape of the distribution, making it generally superior to Top-K for generating natural language. In practice, modern systems often use a combination of Temperature, Top-K, and Top-P together.