Log-Sum-Exp

The trick rests on one identity: $\log \sum_i e^{x_i} = m + \log \sum_i e^{x_i - m}$ where $m$ is the largest $x_i$. Subtracting the maximum makes the biggest exponent exactly $e^0 = 1$, so nothing can overflow, and adding $m$ back afterwards makes the result exact rather than approximate.

Without it, softmax and cross-entropy break on ordinary inputs. A logit of 800 gives $e^{800}$, which is inf in any float type — float32 overflows at about 88 — and inf/inf is nan, which then contaminates every gradient it touches. In the other direction a small probability underflows to zero and log(0) is -inf.

This is why frameworks fuse softmax and cross-entropy into a single operation and expect raw logits rather than probabilities. log_softmax(x) is not log(softmax(x)) in floating point, and hand-assembling the second one is a real and common bug. Use the fused API.