Regression Metrics

Each metric prices mistakes differently. MAE charges every pallet equally while RMSE lets the worst day dominate the score.

Twenty errors of 30 plus one error of 300 give MAE 42.9 and RMSE 71.7, so the lone outlier owns one third of the absolute total but five sixths of the squared total

Why Does This Exist?

A depot forecasts pallets of bottled water. Most days the miss is about 30 pallets. One public holiday the depot opened for an hour, the true demand was near zero, and the model missed by 300.

Two models disagree about which forecast is better. One wins on MAE, the other on RMSE, and someone has to ship one. The metrics disagree because they price that one bad day differently, and choosing is a decision about what a large miss costs.

One idea underneath everything here. An error $e_i$ means prediction minus actual on day $i$ , in pallets, so positive means over-prediction. If you have not priced right against wrong for categories yet, start with classification metrics.

This page covers how MAE, RMSE, MAPE, and R-squared price errors. The line they score lives in linear regression. The geometry behind the fit lives in ordinary least squares.

Think of It Like This

Two ways to write a speeding fine

Schedule A charges 2 pounds for every unit over the limit. Sixty drivers each 1 unit over pay 120 between them. One driver 60 over also pays 120. The schedule cannot tell those apart, and it is not trying to. It prices total drift.

Schedule B charges 2 pounds times the square of the excess. The sixty mild drivers still pay 120 between them. The one driver 60 over now pays 7200, and every enforcement decision under B is about finding that car.

Neither schedule is wrong. Each is a claim about what causes harm. MAE is schedule A and RMSE is schedule B, and picking one means making that claim about your own problem whether you notice or not.

Where it stops: fines are chosen to punish, while metrics are chosen to describe. A fine changes driver behaviour by design, but switching from MAE to RMSE does not change the forecast. It only changes which forecast looks best.

How It Actually Works

Call $e_i$ the error on day $i$ and $n$ the number of days. Both scores below land in pallets, so both read beside a real quantity:

\mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n}\lvert e_i \rvert \qquad \mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} e_i^{2}}

$|e_i|$ is the absolute miss and $e_i^2$ is the squared miss. MAE counts each pallet once. RMSE squares first, averages, then takes the root, so large misses count disproportionately.

Worked example: twenty small misses and one holiday

Twenty days miss by exactly 30 pallets and the holiday misses by 300. The absolute total is:

20 \times 30 + 300 = 900

The squared total is:

20 \times 900 + 90000 = 108000

Divide and root:

\mathrm{MAE} = 900 / 21 \approx 42.9 \qquad \mathrm{RMSE} = \sqrt{108000 / 21} \approx 71.7

Delete that one day and both become exactly 30.0, because equal errors give the squaring nothing to amplify. Of the 900 absolute total the holiday contributes 300, which is one third. Of the 108000 squared total it contributes 90000, which is five sixths. That split is the diagram above and the whole difference between the two.

Why MAPE divides by the actual

MAPE averages each error as a fraction of that day's actual, $|e_i| / |y_i|$ , times 100, where $y_i$ is the true demand. Across the twenty normal days that is about 6 percent. Add the holiday, 300 off a true 4, and that day alone scores 7500 percent, hauling the mean toward 360 percent.

A subtler problem survives even without near-zero days. Under-prediction can be wrong by at most 100 percent, since predicting zero against a true 4 is 100 percent off. Over-prediction has no ceiling, since predicting 100 against a true 4 is 2400 percent off. So MAPE charges over-forecasts harder, and anything tuned on it drifts low.

Why R-squared needs its baseline beside it

R^2 = 1 - \frac{\sum_i (y_i - \hat{y}_i)^2}{\sum_i (y_i - \bar{y})^2}

Here $\hat{y}_i$ is the model prediction and $\bar{y}$ is the mean of the actuals. The numerator is the model squared error and the denominator is the squared error of always predicting the mean. So $R^2$ answers one question: how much better than that mean are you. Zero means no better and one means perfect.

Three consequences get missed. Training $R^2$ never falls when you add a feature, so it proves nothing alone. It is unreadable without the baseline, because the denominator is the baseline. It goes negative on held-out data the moment the model loses to that mean, which is real information that the mean would have won.

Code

The point is the check: rebuild the worked example with plain Python and confirm 42.9 against 71.7.

1def mae(errors: list[float]) -> float:2    return sum(abs(e) for e in errors) / len(errors)3
4
5def rmse(errors: list[float]) -> float:6    return (sum(e * e for e in errors) / len(errors)) ** 0.57
8
9errors: list[float] = [30.0] * 20 + [300.0]10print(f"{mae(errors):.1f} {rmse(errors):.1f}")  # -> 42.9 71.711print(f"{300 / 900:.2f} {90000 / 108000:.2f}")  # -> 0.33 0.83

The first line matches the hand computation. The second line shows the outlier shares: 0.33 is one third of the absolute total and 0.83 is five sixths of the squared total. Remove the last entry and both print 30.0.

Watch Out For

MAPE on a target that touches near zero

Symptom: a dashboard reads above 300 percent and someone asks whether the model broke. It did not. One day had a true value of 4, the model said 304, and dividing by 4 gave 7500 percent for that row. The mean of twenty small percentages and one enormous one is the enormous one with extra steps.

Fix: use MAE, which stays in pallets and calls that day a 300 pallet miss, which it was. If a relative number is genuinely needed across depots of different sizes, divide total absolute error by total actual across all days, which no single row can dominate. Treat closed days as missing by decision, not as outliers to average through.

Comparing R-squared across two different datasets

Symptom: depot A scores 0.91, depot B scores 0.62, and the write-up concludes the model works better at A. But A serves a town with wild seasonal swings, so its actuals vary enormously and its denominator is huge. B is steady. The model may be more accurate at B in pallets, and $R^2$ cannot tell you because the denominator moved.

Fix: compare MAE or RMSE in units when the datasets differ, and keep $R^2$ inside one fixed dataset. Always put a baseline beside the model score, such as always predicting the mean from train validation and test splits.

The Quick Version

MAE is the mean absolute miss in target units. A 10 unit miss counts ten times a 1 unit miss.
RMSE squares misses first, so the worst prediction dominates. Twenty misses of 30 plus one of 300 give MAE 42.9 and RMSE 71.7.
The holiday owns one third of the absolute total and five sixths of the squared total, which is why RMSE reacts to large misses and MAE does not.
MAPE divides by the actual, so it explodes near zero and punishes over-prediction harder than under-prediction.
R-squared measures improvement over always predicting the mean. It never falls from adding a training feature and can go negative on held-out data.
Report MAE by default, add RMSE when large misses cost extra, and never publish either without a baseline beside it.