Softmax

Softmax exponentiates each score and divides by the total: $\sigma(z)_i = e^{z_i} / \sum_j e^{z_j}$. Exponentiating forces everything positive, and dividing by the sum forces the total to one, so the raw scores from a network's final layer — the logits — become a usable distribution.

Because the total is fixed at one, raising one class's probability necessarily lowers the others. That is a property of the function, not a claim about the world, and it is why softmax suits single-label problems and not multi-label ones, where independent sigmoids are correct instead.

Adding the same constant to every logit leaves the output unchanged, which is both why softmax is shift-invariant and how it is implemented safely: subtract the maximum logit before exponentiating. Skipping that step overflows at a logit of about 88 in float32, giving inf, then nan. Dividing the logits by a temperature before applying softmax sharpens the distribution below one and flattens it above.