Cross-Entropy

Why Does This Exist?

Classification needs a loss that punishes confident mistakes far harder than hesitant ones, and accuracy cannot do it. A model that says 51 percent for the right class and one that says 99 percent both count as correct and score identically. There is no gradient to follow, so there is nothing to train on.

Cross-entropy fixes exactly this. A prediction of $0.99$ on the right answer costs $-\ln(0.99) \approx 0.01$. A prediction of $0.01$ on it costs $-\ln(0.01) \approx 4.6$, over four hundred times as much. Being confidently wrong is expensive, being tentatively wrong is survivable, and the resulting slope points somewhere useful.

It is also the principled choice rather than a convenient one: minimising cross-entropy is mathematically identical to maximum likelihood estimation under a categorical assumption. It is not a heuristic that happens to work. It is what the probability theory tells you to do.

Think of It Like This

How It Actually Works

Cross-entropy measures the average surprise of using distribution $q$ to describe outcomes actually drawn from $p$:

In plain words: weight the surprise of each outcome under your predicted distribution $q$ by how often it truly happens under $p$. Compare entropy with $H(p) = -\sum p \log p$: the only change is that the logarithm is taken of $q$ rather than $p$. Same average surprise, computed with the wrong probabilities.

Why the formula collapses to one term

For classification the true distribution is one-hot: the correct class has probability $1$ and every other class has $0$. Every term where $p(x_i) = 0$ vanishes, and the single surviving term has $p = 1$:

In plain words: the loss is the negative log of the probability assigned to the right answer, full stop. Wrong-class probabilities never appear in the formula. They matter only indirectly, because the softmax forces probabilities to total $1$, so raising a wrong class lowers the right one.

Its relationship to entropy and KL divergence

Cross-entropy decomposes cleanly:

In plain words: the cost of using $q$ splits into the uncertainty genuinely present plus the extra paid for $q$ being wrong. Since $H(p)$ depends only on the data and not on the model, minimising cross-entropy and minimising KL divergence are the same optimisation.

This also explains why the loss rarely reaches zero on real data. Its floor is $H(p)$, the irreducible uncertainty in the labels. If two identical inputs carry different labels, no model drives below that, and chasing zero means memorising noise.

The gradient is the reason it is used

Pair a softmax output with cross-entropy and the gradient with respect to the pre-softmax logits $z$ is startlingly simple:

In plain words: predicted probabilities minus the one-hot target. That is the entire backward pass for the output layer. The gradient scales with the error, so badly wrong predictions produce large gradients and learning is fast where needed. No derivative of the exponential survives, which is why the two are always implemented as one fused operation.

Worked example

Three classes, the true class is the second one, so $y = [0, 1, 0]$. The model predicts $\hat{y} = [0.1, 0.7, 0.2]$.

Only the true class contributes:

Hand check: $\ln(0.7) \approx -0.3567$, so negating gives $0.3567$, rounded to $0.357$. With a worse prediction $\hat{y} = [0.2, 0.2, 0.6]$ on the same label:

Hand check: $\ln(0.2) \approx -1.6094$, so the loss is $1.6094$. Four and a half times the first loss from the same label. The gradient in the first case is $\hat{y} - y = [0.1, -0.3, 0.2]$: push the true class up by $0.3$ and push the wrong classes down. A useful reference: a uniform guess over $3$ classes gives $-\ln(1/3) \approx 1.099$. Any loss above that means the model does worse than knowing nothing.

Code

The snippet below uses only the standard library. It recomputes both hand losses and shows the stable logit path that survives large magnitudes.

1import math2
3
4def cross_entropy_from_logits(logits: list[float], target: int) -> float:5    # Log-sum-exp: subtract the max first so exp() can never overflow.6    m = max(logits)  # softmax is unchanged by shifting every logit7    shifted = [v - m for v in logits]8    log_sum = math.log(sum(math.exp(v) for v in shifted))9    log_probs = [v - log_sum for v in shifted]  # log softmax, stable10    return -log_probs[target]11
12
13print(round(-math.log(0.7), 4))  # -> 0.3567  the true class14print(round(-math.log(0.2), 4))  # -> 1.6094  confidently wrong15print(round(-math.log(1.0 / 3.0), 4))  # -> 1.0986  uniform over 316
17logits = [800.0, 802.0, 801.0]  # naive math.exp(800) overflows here18print(round(cross_entropy_from_logits(logits, 1), 4))  # -> 0.407619try:20    naive = [-math.log(math.exp(v) / sum(math.exp(u) for u in logits)) for v in logits]21    print(naive[1])22except OverflowError as e:23    print(type(e).__name__)  # -> OverflowError

The first three prints match the hand arithmetic: $0.357$, $1.609$, and $1.099$ for uniform. The last pair is the point: identical mathematics, and only the shifted version survives realistic logit magnitudes because the largest exponent becomes $e^0 = 1$.

Watch Out For

The Quick Version

• Cross-entropy is the average surprise of describing reality $p$ with prediction $q$: $-\sum p \log q$.
• With one-hot targets it collapses to $-\log q(\text{correct class})$. Wrong-class probabilities never appear directly.
• Confidently wrong is punished enormously: $0.7$ costs $0.357$ while $0.2$ costs $1.609$, and zero on the truth costs infinity.
• It decomposes as $H(p) + D_{\mathrm{KL}}(p \parallel q)$, so minimising it is minimising KL divergence.
• Its floor is $H(p)$, not zero. A uniform guess over 3 classes gives $1.099$.
• Paired with softmax, the gradient on the logits is just $\hat{y} - y$, which is why they are fused.
• Minimising cross-entropy is maximum likelihood under a categorical assumption.
• Always compute from logits with log-sum-exp. Separate softmax and log overflows on ordinary magnitudes.

What to Read Next

• Derivatives and Partial Derivatives
• Descriptive Statistics