Skip to content
AI360XpertBeta

Forecast Evaluation

A forecast is only as good as the ruler used to grade it. Different error metrics punish the same miss in very different ways.

Errors 2 and 10 average to 6 under MAE but square to 4 and 100 under RMSE for 7.21, so one reads 5 times worse and the other 25 times worse
Errors 2 and 10 average to 6 under MAE but square to 4 and 100 under RMSE for 7.21, so one reads 5 times worse and the other 25 times worse

Why Does This Exist?

Predict 100100 sales and sell 9090 and the miss is 1010. Predict 1010 and sell 00 and the miss is also 1010. The first is a 1010 percent slip. The second stocks a shelf nobody touches.

A single raw gap cannot tell those apart. Teams pick a metric that matches what hurts: steady misses, rare disasters, percentages for planning, or proof against doing nothing. The wrong ruler picks the wrong model.

This page covers point forecast rulers. The models being graded live in ARIMA and exponential smoothing. Honest testing over time lives in backtesting. Scoring classifiers instead lives in classification metrics.

Think of It Like This

A teacher grading two quizzes

A teacher grades the same two wrong answers three ways. Under the flat rule each wrong answer loses one point, so a small slip and a huge blunder differ little. Under the square rule a huge blunder loses far more than several small slips combined. Under the percent rule a miss on a ten point question hurts more than the same miss on a hundred point question.

Call the error ee the gap between forecast and actual. Flat grading is MAE, square grading is RMSE, percent grading is MAPE. Each teacher passes a different student from the same papers.

Where it stops: teachers can excuse a blank page. These formulas grade every number you hand them, including zeros that break percent math and gaps that hide direction.

How It Actually Works

Call actual yy and forecast y^\hat{y} for one point. Call error e=y^−ye = \hat{y} - y. With nn points, mean absolute error averages the absolute gaps, root mean squared error roots the averaged squares, mean absolute percentage error averages gaps relative to actuals, and mean absolute scaled error scales against a naive one step repeat:

MAE=1n∑∣e∣,RMSE=1n∑e2\text{MAE} = \frac{1}{n} \sum \lvert e \rvert, \quad \text{RMSE} = \sqrt{\frac{1}{n} \sum e^{2}} MAPE=100n∑∣ey∣,MASE=MAEMAE of naive\text{MAPE} = \frac{100}{n} \sum \left\lvert \frac{e}{y} \right\rvert, \quad \text{MASE} = \frac{\text{MAE}}{\text{MAE of naive}}

MAE treats misses linearly and resists outliers. RMSE squares first, so one large miss dominates. MAPE speaks in percent but divides by yy, so y=0y = 0 breaks it and over forecasts score differently than under forecasts. MASE divides by the naive repeat forecast, so above 1.01.0 means a simple repeat wins. That ladder is why backtesting reports more than one number.

Worked example: misses of 2 and 10

Take two points with absolute errors 22 and 1010. MAE is:

(2+10)/2=6(2 + 10) / 2 = 6

Squared errors are 22=42^2 = 4 and 102=10010^2 = 100. Mean square is (4+100)/2=52(4 + 100) / 2 = 52. RMSE is:

52≈7.21\sqrt{52} \approx 7.21

The 1010 looks five times the 22 under MAE since 10/2=510 / 2 = 5. It looks twenty five times under squaring since 100/4=25100 / 4 = 25. That five against twenty five is the diagram headline: two equal bars on the left, one pair growing linearly in the middle, one pair exploding on the right.

What percent and scaled add

MAPE on y=100y = 100 missing by 1010 reads 1010 percent, which planners quote directly. On y=0y = 0 the same 1010 divides by zero and the formula fails, so teams clip, skip zeros, or switch metric. MASE answers a harder question: with MAE 66 against naive MAE 88, MASE is 6/8=0.756 / 8 = 0.75, below 1.01.0, so the model beats doing nothing. Above 1.01.0, ship the naive repeat from exponential smoothing instead.

Code

Rebuild the worked example with plain Python and confirm 6.06.0 and 7.217.21.

import math
errors: list[float] = [2.0, 10.0]mae: float = sum(abs(e) for e in errors) / len(errors)mse: float = sum(e * e for e in errors) / len(errors)rmse: float = math.sqrt(mse)
print(f"{mae:.1f} {rmse:.2f}")  # -> 6.0 7.21print(f"{10.0 / 2.0:.0f} {(10.0**2) / (2.0**2):.0f}")  # -> 5 25
actual: list[float] = [100.0, 150.0, 200.0, 50.0]forecast: list[float] = [110.0, 140.0, 180.0, 90.0]mae4: float = sum(abs(f - a) for a, f in zip(actual, forecast)) / len(actual)print(f"{mae4:.1f}")  # -> 20.0

The printout matches the hand arithmetic. Change the 1010 to 2020 and watch RMSE jump far faster than MAE, which is the square penalty in numbers.

Watch Out For

Quoting MAPE with zeros in the actuals

Percent error divides by actual, so one zero row makes the average infinite or silently huge.

Symptom: MAPE reads thousands of percent while MAE and RMSE look calm. Fix: filter or clip zeros before scoring, or report MASE from this page alongside backtesting windows that show where zeros cluster.

Tuning to RMSE while the business pays MAE

RMSE pushes the model to avoid rare large misses at the cost of slightly larger everyday misses. When stock cost is linear, that tradeoff loses money.

Symptom: RMSE improves release over release while holding cost or MAE stalls. Fix: pick the ruler from the cost first, as in classification metrics for classifiers, and keep the other metric as a diagnostic only.

The Quick Version

  • Forecast evaluation picks the ruler that matches the business cost of being wrong.
  • Errors 2 and 10 average to MAE 6.0 in our example.
  • The same errors square to 4 and 100, mean 52, and root to RMSE 7.21.
  • A miss of 10 reads 5 times a miss of 2 under MAE and 25 times under squaring.
  • MAPE speaks in percent but fails when any actual equals zero.
  • MASE below 1.0 proves the model beats a naive repeat, while above 1.0 favors the naive rule.