Softmax Function and its Derivative

Raw scores can be any numbers, while probabilities must be positive and sum to 1. Softmax bridges the gap by exponentiating each score and dividing by the total.

$Softmax turns logits 2, 1, 0 into probabilities 0.665, 0.245, 0.090 that sum to 1, giving the largest logit the largest share$

Softmax turns logits 2, 1, 0 into probabilities 0.665, 0.245, 0.090 that sum to 1, giving the largest logit the largest share

Why Does This Exist?

Classifiers output raw scores called logits: any real numbers, in any range, with no promise they sum to anything. Training and sampling need probabilities instead: positive numbers that sum to $1$ . Normalizing by the sum fails the moment scores go negative, and clipping breaks gradients.

Softmax fixes the mismatch in one smooth step that keeps order, stays differentiable, and pairs cleanly with the standard classification loss. This page covers the function and its derivative. The loss it feeds lives in cross-entropy and common loss functions. A cousin that turns distances into probabilities appears in t-SNE and UMAP.

The one idea you need first

A logit is an unconstrained score $z_i$ , while a probability $p_i$ must satisfy $p_i \ge 0$ and $\sum_i p_i = 1$ . Exponentiation maps any real to a positive number, and dividing by the total forces the sum to $1$ . Those two moves are the whole function.

Think of It Like This

Splitting a prize by loudness of applause

Three street performers earn applause measured at levels $2$ , $1$ , and $0$ . You must split $100 in proportion to enthusiasm, but applause levels can be negative on a bad night and they never sum to $100$ .

So you convert first: exponentiate each level to get positive tickets, then share the prize by ticket counts. The loudest act gets the most tickets and the largest share, yet nobody gets everything. Louder overall applause does not change the split, only the gaps between acts do.

Where it stops: prize shares are spent once, while softmax outputs are reused as weights that blend values downstream. The split is the start of computation, not the payout.

How It Actually Works

For logits $z_1, z_2, \ldots, z_n$ , where each $z_i$ is one raw score and $n$ counts classes:

\sigma(z)_i = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}

In plain words: exponentiate each score with $e^{z_i}$ , sum those positives, then divide each by the sum. Outputs are positive and sum to $1$ by construction. Shifting all logits by the same constant cancels out, so only gaps matter. Large gaps concentrate mass, which is why attention blends rather than selects until gaps grow wide.

Why the derivative is a matrix

Softmax outputs share one denominator, so nudging $z_k$ moves every $p_i$ , not just $p_k$ . Its Jacobian $J$ , where $J_{ik} = \partial p_i / \partial z_k$ and $p_i$ is the $i$ -th output, is:

J_{ik} = p_i (\delta_{ik} - p_k)

Here $\delta_{ik}$ is $1$ when $i = k$ and $0$ otherwise. Diagonal terms $p_i(1 - p_i)$ say each output rises with its own logit but saturates near $0$ or $1$ . Off-diagonal terms $-p_i p_k$ say raising one logit steals share from the rest. Rows sum to zero because total mass is conserved.

Paired with cross-entropy loss $L = -\log p_y$ for the true class $y$ , the chain collapses to $p - y$ , where $y$ is one-hot. That clean gradient is why the pair dominates classification.

Worked example

Take $z = [2, 1, 0]$ .

Exponentiate: $e^2 \approx 7.39$ , $e^1 \approx 2.72$ , $e^0 = 1.00$ . Sum to $7.39 + 2.72 + 1.00 = 11.11$ . Divide:

p = [7.39/11.11, 2.72/11.11, 1.00/11.11] \approx [0.665, 0.245, 0.090]

The top score takes two-thirds of the weight, not all of it. Sum check: $0.665 + 0.245 + 0.090 = 1.000$ .

Jacobian diagonal for the first output is $0.665 \times (1 - 0.665) \approx 0.223$ . Off-diagonal stealing from the second is $-0.665 \times 0.245 \approx -0.163$ . Every row sums to zero up to rounding, confirming mass conservation.

Code

The point is stability, not the formula retyped. Subtracting the max before exponentiating prevents overflow without changing the result.

1import numpy as np2
3
4def softmax(z: np.ndarray) -> np.ndarray:5    e = np.exp(z - z.max(axis=-1, keepdims=True))  # shift so max is 0, exp cannot overflow6    return e / e.sum(axis=-1, keepdims=True)7
8
9z = np.array([2.0, 1.0, 0.0])10print(np.round(softmax(z), 3))  # -> [0.665 0.245 0.09 ]11print(float(softmax(z).sum()))  # -> 1.0

Outputs $[0.665, 0.245, 0.090]$ sum to $1.0$ , exactly the hand computation. Without the shift, $e^{1000}$ overflows to infinity on the same inputs shifted up.

Watch Out For

Exponentiating without subtracting the max

Calling np.exp(z) directly overflows once logits pass about $709$ in float64 and $88$ in float32, returning infinity and then NaN probabilities. Large logits are normal before training settles, so this breaks early runs rather than edge cases.

Symptom: loss turns NaN while logits look merely large. Fix: always subtract the row max first, as the snippet does. The math is identical because the shift cancels in numerator and denominator.

Reading softmax outputs as calibrated confidence

Softmax must sum to $1$ , so it always looks confident even on garbage inputs. A $[0.665, 0.245, 0.090]$ split on unfamiliar data does not mean 66.5 percent accuracy. Temperature scaling, label smoothing, and calibration checks exist because raw shares mislead.

Symptom: high-confidence errors on shifted data, worse when gaps are large. Fix: treat outputs as weights for blending or ranking, calibrate before quoting confidence, and never use argmax probabilities as reliability scores without validation.

The Quick Version

Softmax maps any real logits to positive probabilities that sum to $1$ via exponentiate then normalize.
Only gaps between logits matter: shifting all scores leaves every output unchanged.
On $[2, 1, 0]$ , exponentials $[7.39, 2.72, 1.00]$ sum to $11.11$ and give $[0.665, 0.245, 0.090]$ .
The Jacobian is $p_i(\delta_{ik} - p_k)$ : each output steals share from the rest, and rows sum to zero.
With cross-entropy, the gradient collapses to $p - y$ , which is why the pair trains well.
Always subtract the max before exp, and never read raw shares as calibrated confidence.