Common Loss Functions

One number that says how wrong a prediction is. Squared error punishes big misses hard, absolute error stays steady, and log loss punishes confident wrong classifiers.

Squared error grows quadratically with the miss while absolute error grows linearly, so both equal 1 at an error of 1 but squared error dominates beyond it

Why Does This Exist?

A model cannot improve until "wrong" becomes a number. Training is minimisation, and minimisation needs a single scalar to push down. The loss function is that scalar: it takes a prediction and a target and returns one number measuring the miss.

Different tasks need different measures. Predicting a house price punishes being 100 thousand off far more than being 10 thousand off, and squared error encodes exactly that. Predicting a class probability punishes confident wrongness, and log loss encodes that instead. The choice changes the gradients, the optimum, and the robustness, which is why the loss is a modelling decision rather than plumbing.

This page catalogues the three losses you will meet first. The goal-plus-constraints framing around them is objective functions and constraints. Why log loss is the same as maximum likelihood is maximum-likelihood estimation. What consumes the loss to take a step is gradients.

The one idea you need first

A residual is prediction minus target, written $e = \hat{y} - y$ . Every regression loss below is a function of $e$ : it maps the miss to a penalty. Classification losses instead take a predicted probability $p$ for the true class.

Think of It Like This

A taxi meter, a flat fine, and a confident bet

Squared error is a taxi meter that accelerates: the first mile off route costs a little, each extra mile costs more than the last. A miss of 2 costs 4, a miss of 3 costs 9. Outliers run up the bill fast, which is why one bad reading can drag the whole fit toward itself.

Absolute error is a flat fine per mile: every extra unit of miss costs the same. A miss of 2 costs 2, a miss of 3 costs 3. Outliers still pay, but they cannot hijack the total.

Log loss is a confident bet: predicting the true class with probability 0.99 when wrong costs little if right and a fortune if wrong. Confidence is priced, which is exactly what a classifier should feel.

Where the analogy stops: meters never reward, but some training objectives add bonuses for margin. Losses here only penalise.

How It Actually Works

Three formulas, three behaviours. Let $y$ be the target, $\hat{y}$ the prediction, and $e = \hat{y} - y$ .

Mean squared error (MSE) averages $e^2$ over examples. For one example with $y = 1$ and $\hat{y} = 1.3$ , the error is $e = 0.3$ and the loss is $0.3^2 = 0.09$ . Squaring makes the gradient proportional to the error, so large misses pull harder. The price is outlier sensitivity: a miss of 10 contributes 100.

Mean absolute error (MAE) averages $|e|$ . The same example gives $|0.3| = 0.3$ . Its gradient has constant size, so every example pulls equally hard regardless of miss size. That resists outliers but the kink at zero means no unique derivative there, and optimisation creeps rather than settles.

Binary cross-entropy (log loss) scores a predicted probability $p$ for the true class as $-\log(p)$ . Predicting the true class with $p = 0.7$ gives $-\log(0.7) \approx 0.3567$ . Predicting with $p = 0.99$ gives $-\log(0.99) \approx 0.0101$ , while $p = 0.01$ gives $-\log(0.01) \approx 4.6052$ . Confident wrong answers explode, which is the behaviour a probability model needs.

Worked example

True label $y = 1$ , predicted probability $p = 0.7$ , point prediction $\hat{y} = 0.7$ .

Error: $e = 0.7 - 1 = -0.3$ , magnitude 0.3.
Squared error: $(-0.3)^2 = 0.09$ .
Absolute error: $|-0.3| = 0.3$ .
Cross-entropy: $-\log(0.7)$ . Since $\log(0.7) \approx -0.3567$ , the loss is about $0.3567$ .

At error magnitude 1 the two regression losses meet: squared gives 1, absolute gives 1, which is the crossover the diagram marks. Below 1, squaring shrinks the penalty (0.09 versus 0.3). Above 1 it amplifies it: at error 2, squared gives 4 while absolute gives 2.

Code

The point is the comparison on one miss: the same error feeds all three formulas, so their different appetites show directly.

1import math2
3y, y_hat, p = 1.0, 0.7, 0.74e = y_hat - y5mse = e**26mae = abs(e)7bce = -math.log(p)8
9print(round(e, 4))    # -> -0.310print(round(mse, 4))  # -> 0.0911print(round(mae, 4))  # -> 0.312print(round(bce, 4))  # -> 0.3567

Minus 0.3 becomes 0.09 under squaring, stays 0.3 under absolute value, and the probability 0.7 costs 0.3567 under log loss, matching the hand computation.

Watch Out For

Training regression with MSE on outlier-heavy targets

The symptom is a model that chases a handful of extreme points while fitting the bulk poorly. One target recorded 100 times too large contributes $100^2 = 10{,}000$ times the penalty of a unit miss, so its gradient drowns out hundreds of normal examples.

Fix: inspect the target distribution first, then switch to MAE or Huber, cap or clean the extremes, or model log-targets. Never average squared errors blindly on heavy-tailed data.

Feeding logits into a probability loss without the sigmoid

The symptom is a loss that is negative or explodes, plus gradients pointing nowhere. Cross-entropy needs $p$ in $(0, 1)$ , but the model outputs an unbounded logit $z$ . Passing $z = 2.0$ directly as $p$ gives $-\log(2.0) \approx -0.6931$ , a negative "loss".

Fix: squash first with $p = 1 / (1 + \exp(-z))$ , or call the combined logits variant your library ships. For $z = 2.0$ , $p \approx 0.8808$ and the loss is $-\log(0.8808) \approx 0.1270$ .

The Quick Version

A loss turns a wrong prediction into one number that training minimises.
Squared error squares the miss: 0.3 becomes 0.09, 2 becomes 4. Outliers dominate.
Absolute error keeps the miss as is: steady gradients, outlier-resistant, kinky at zero.
Cross-entropy charges $-\log(p)$ : predicting 0.7 when right costs about 0.3567, predicting 0.01 costs about 4.6052.
Below error 1, squared error is the smaller penalty. Above 1, it is the larger one.
Heavy-tailed targets want MAE or Huber, never blind MSE.
Squash logits to probabilities before applying log loss.