Perplexity

Why Does This Exist?

You train a model with next-token prediction and at every position it hands you a full distribution over the vocabulary. Staring at a few samples tells you nothing, but shipping without a number is worse. You need one score that says how well those distributions matched real text the model never trained on.

Perplexity is that score, and it's not a new invention. It is the training loss itself, reshaped into units you can picture. Where cross-entropy reports surprise in nats, a log-scale unit nobody feels, perplexity converts it into an effective headcount of options. A score of 1.62 says the model behaved as if it hesitated among fewer than two live candidates per step. A score of 50000 says it guessed blindly across a 50000 word vocabulary.

The one idea you need first is the negative log-probability. If the model gives the true next token probability $p$, its surprise at that position is $-\log p$. Right and confident gives a tiny number, wrong or timid gives a large one. Perplexity just averages those surprises and undoes the log. This page covers that intrinsic number only. Task skill and factuality live in post-training evaluation, and confidence quality lives in LLM calibration.

Think of It Like This

How It Actually Works

Take held-out text with $n$ next-token positions. At position $t$, the model sees the context $x_1, \ldots, x_t$ and assigns some probability to whatever token actually came next, written $p_\theta(x_{t+1} \mid x_1, \ldots, x_t)$. Here $n$ counts scored positions, $x$ is a token, and $p_\theta$ is your trained model with weights $\theta$. The log is the natural log, so surprise is measured in nats.

Perplexity averages the surprise, then exponentiates:

In plain words: read off the probability given to each true token, take the negative log of each, average, and raise $e$ to that average. If you use log base 2 instead, you get the same idea in bits, and perplexity equals $2$ raised to the average bits. Same score, different ruler.

Why the exponent matters

Cross-entropy already averages surprise, so why add the $\exp$? Because logs compress differences your intuition needs. An average surprise of 0.48 nats versus 2.99 nats looks like a small gap on paper, but the first means fewer than two live options per step while the second means twenty. Exponentiating restores the linear headcount. That's the whole trick, and it is why a drop from 20 to 10 matters far more than a drop from 110 to 100, even though both shave the same 10 points.

Worked example: four tokens score 1.62

Say your model reads four unseen positions and gives the true next tokens probabilities $[0.80, 0.50, 0.90, 0.40]$. You have seen nothing else. That list is the entire fixture, and the diagram above draws exactly these four bars.

Turn each into surprise with $-\log p$, rounded to two places:

• $-\log(0.80) = 0.22$
• $-\log(0.50) = 0.69$
• $-\log(0.90) = 0.11$
• $-\log(0.40) = 0.92$

Add them: $0.22 + 0.69 + 0.11 + 0.92 = 1.94$ displayed, $1.9379$ before rounding. Divide by $n = 4$ positions: $1.9379 / 4 = 0.4845$, shown as $0.48$ nats. Exponentiate: $\exp(0.4845) \approx 1.62$.

So this model scores about 1.62. Read it as a headcount: at each step it behaved as if torn among roughly 1.6 equally likely tokens. For contrast, a model giving $0.05$ to every true token scores $\exp(-\log 0.05) = 20.00$, or twenty live options. Same formula, far more confusion.

What the number means, and what it costs

A perplexity of $1$ means the model gave probability $1$ to every true token, perfect and never observed on real text. A perplexity near the vocabulary size $V$ means guessing uniformly, the floor you must beat. Real models sit between, and lower is better only when everything else is fixed: same tokenizer, same test text, same token count. For background on surprise itself, see entropy.

The trade is scope for convenience. You get a cheap single number straight from the loss, no judges or tasks needed, but you buy silence on everything beyond pattern match. Perplexity assumes the held-out text looks like the world you care about, that you scored enough tokens for the mean to settle, and that no test sentence leaked into training. Don't use it to judge truth, helpfulness, or style. A model can predict fluent nonsense with total confidence and score beautifully, which is exactly why teams pair it with task tests and contamination checks such as benchmark contamination.

Code

This snippet recomputes the fixture above, so you can see the average-then-exponentiate step instead of taking it on faith.

1import numpy as np2
3
4def perplexity_from_probs(probs: list[float]) -> float:5    """Perplexity from the true-token probs at each position."""6    arr = np.asarray(probs, dtype=float)7    avg_neg_log_prob = float(-np.log(arr).mean())8    return float(np.exp(avg_neg_log_prob))9
10
11fixture = [0.80, 0.50, 0.90, 0.40]12print(f"{perplexity_from_probs(fixture):.2f}")  # -> 1.6213print(f"{perplexity_from_probs([0.05, 0.05, 0.05, 0.05]):.2f}")  # -> 20.00

The first line prints $1.62$, the number the worked example derived by hand. The second shows a uniformly confused model at $20.00$, since $1 / 0.05 = 20$. Change one input probability and rerun: you'll feel how harshly a single near-zero probability punishes the mean.

Watch Out For

The Quick Version

• Perplexity is exponentiated average surprise: $\exp$ of the mean $-\log p$ over true next tokens on unseen text.
• Read the result as a headcount of live options per step, where 1 is perfect and the vocabulary size means blind guessing.
• Four true-token probabilities of 0.8, 0.5, 0.9 and 0.4 average to 0.48 nats of surprise and score about 1.62.
• Perplexity inherits the training loss, so it is cheap and intrinsic, with no say on truth or task skill.
• Scores compare only under the same tokenizer and test set, since token splits reshape the average.

What to Read Next

• Cross-Entropy
• Next-Token Prediction
• LLM Confidence & Calibration
• Post-Training Evaluation