Loss Functions

Every common loss is the negative log-likelihood of some noise model, so picking one is a claim about how your data got its errors. Choose the claim first.

One residual of 9.0 owns 78.3 percent of the squared gradient but 10.0 percent of the absolute and 28.6 percent of the Huber gradient

Why Does This Exist?

You predict next month repair bills. Most land between $200 and $800. One van in three hundred needs a $9,000 engine rebuild. You minimise squared error because it is default, and every quote comes back high.

Nothing broke. Squared error says a residual of $8,000$ is ten thousand times more surprising than $80$ . Given that belief, chasing the rebuild is correct.

Every common loss is the negative log likelihood of some noise model. Picking one picks what you believe about errors. Maximum likelihood picks parameters making observed data most probable. Probabilities multiply and underflow, so take logs and flip sign. This page turns that list into a decision. How the chosen loss gets minimised lives in gradient descent. How slopes reach weights lives in backpropagation. How to report what remains lives in regression metrics.

Think of It Like This

Sliding the pivot under a loaded plank

Ten people stand on a plank and you slide a pivot under it.

Balance it and the pivot lands at centre of mass. Pull grows with distance, so the far end drags it. That is squared error, and the pivot is the mean.

Ignore distance instead. Slide until five stand each side, then stop. The far person counts as one like everyone near middle. That is absolute error, and the pivot is the median.

Cap the pull. It grows with distance up to a metre out, then holds flat. Near pivot it balances, far out it counts once. That is Huber.

Where it stops: planks explain centre but not learning. Real losses also set gradient size and differentiability, which wood and weight cannot show.

How It Actually Works

Every loss is a claim about noise

Squared error is $\tfrac{1}{2}(y - \hat{y})^2$ with truth $y$ and prediction $\hat{y}$ . It is the negative log of a normal density plus a constant, so it assumes constant variance Gaussian noise. Slope is the residual itself. A row off by $9.0$ pushes ninety times harder than one off by $0.1$ .

Absolute error is $|y - \hat{y}|$ , the negative log of a Laplace model with heavier tails. Slope is $+1$ or $-1$ only. Distance stops buying influence.

Huber switches at threshold $\delta$ . Squared inside, absolute outside. It assumes normal in the middle and exponential tails beyond, which fits most cost data. You keep a smooth centre and a bounded tail. Price is one more number in target units.

Asymmetry and counts need their own claims. Quantile loss charges sides differently, so at $\tau = 0.9$ under prediction costs nine times over prediction. Poisson and Tweedie cover counts and costs piled at zero with a positive tail. Focal loss reweights, not a noise model. It scales cross entropy by how badly predicted each row is, so a confident majority row barely contributes.

Worked example with nine plus one

Residuals are $[0.2, -0.4, 0.1, 0.3, -0.2, 0.5, -0.1, 0.4, -0.3, 9.0]$ . Last is the engine. Mean is $9.5 / 10 = 0.95$ . Sorted middle pair $0.1$ and $0.2$ average to median $0.15$ .

Gradient share is the worst row pull over total pull. Squared pulls are absolute residuals summing to $11.5$ , so $9.0 / 11.5 \approx 78.3$ percent. Absolute pulls are all $1$ , summing to $10$ , so $1 / 10 = 10.0$ percent. Huber at $\delta = 1.0$ clips to $[0.2, -0.4, 0.1, 0.3, -0.2, 0.5, -0.1, 0.4, -0.3, 1.0]$ in absolute value summing to $3.5$ , so $1.0 / 3.5 \approx 28.6$ percent. One row steers four fifths of a squared update and one tenth of an absolute update. That split is the diagram headline.

Cross entropy and its gradient

Classification has no Gaussian residual, only a category and odds. Cross entropy is the negative log likelihood of that outcome. Take raw score $z$ before squash, push through sigmoid or softmax to $p$ , measure against label $y$ :

\frac{\partial L}{\partial z} = p - y

Predicted minus actual. No leftover squash derivative. The sigmoid slope cancels with the log exactly. A confident wrong prediction gives gradient near $1$ , not near zero as squared error on a saturated sigmoid would, which is why logistic regression uses this loss.

A loss must be differentiable, a metric need not

A loss is what the optimiser descends, so slopes must flow. A metric is what you report and can be anything. Accuracy is the clean split. Nudge a weight and no prediction crosses threshold, so gradient is zero. Push far and one flips, so accuracy jumps. Nothing to descend, which is why nothing trains on it.

Code

Nine ordinary residuals and one rebuild. What share does that row ask.

1resids: list[float] = [0.2, -0.4, 0.1, 0.3, -0.2, 0.5, -0.1, 0.4, -0.3, 9.0]2
3def share(pulls: list[float]) -> float:4    total: float = sum(abs(g) for g in pulls)5    return max(abs(g) for g in pulls) / total6
7squared: list[float] = resids8absolute: list[float] = [1.0 if r > 0 else -1.0 for r in resids]9huber: list[float] = [max(-1.0, min(1.0, r)) for r in resids]10
11print(f"{share(squared):.1%}")  # -> 78.3%12print(f"{share(absolute):.1%}")  # -> 10.0%13print(f"{share(huber):.1%}")  # -> 28.6%

The printout matches the hand division. Change $9.0$ to $0.9$ and squared share falls below $30$ percent. The loss chooses the outlier weight, not the data.

Watch Out For

Squared error on a heavy tailed target

Predictions run high across bulk data and low on extremes. Mean error near zero, median clearly positive. Squared error does what it was asked with a few rows owning the gradient.

Symptom: residual plot fans wider with level plus one extreme row owning most gradient. Fix: switch to Huber with $\delta$ near a high percentile of ordinary residuals, model log target and accept a median, or use Tweedie for costs at zero. Do not delete extremes. Those vans exist.

Optimising a loss no decision uses

Training loss improves weekly while the reviewed number sits still. Notebook progress never lands downstream.

Symptom: loss and decision disagree on what matters, often a ranked shortlist where middle accuracy never reaches Monday bookings. Fix: train the decision directly, such as will this van land in worst ten percent, and report it with regression metrics rather than polishing an unused loss.

The Quick Version

Every common loss is the negative log likelihood of a noise model about errors.
Our ten residuals average 0.95 but median 0.15 because one 9.0 drags the mean.
That row owns 78.3 percent of a squared update, 10.0 percent of absolute, 28.6 percent of Huber.
Cross entropy is the classification default with gradient predicted minus actual on raw scores.
Quantile prices asymmetry, focal down weights confident majorities, Tweedie handles zero piled costs.
A loss must be differentiable. A metric need not, which is why nothing trains on accuracy.