Forecast Evaluation
A forecast is only as good as the ruler used to grade it. Different error metrics punish the same miss in very different ways.
Why Does This Exist?
Predict sales and sell and the miss is . Predict and sell and the miss is also . The first is a percent slip. The second stocks a shelf nobody touches.
A single raw gap cannot tell those apart. Teams pick a metric that matches what hurts: steady misses, rare disasters, percentages for planning, or proof against doing nothing. The wrong ruler picks the wrong model.
This page covers point forecast rulers. The models being graded live in ARIMA and exponential smoothing. Honest testing over time lives in backtesting. Scoring classifiers instead lives in classification metrics.
Think of It Like This
A teacher grading two quizzes
A teacher grades the same two wrong answers three ways. Under the flat rule each wrong answer loses one point, so a small slip and a huge blunder differ little. Under the square rule a huge blunder loses far more than several small slips combined. Under the percent rule a miss on a ten point question hurts more than the same miss on a hundred point question.
Call the error the gap between forecast and actual. Flat grading is MAE, square grading is RMSE, percent grading is MAPE. Each teacher passes a different student from the same papers.
Where it stops: teachers can excuse a blank page. These formulas grade every number you hand them, including zeros that break percent math and gaps that hide direction.
How It Actually Works
Call actual and forecast for one point. Call error . With points, mean absolute error averages the absolute gaps, root mean squared error roots the averaged squares, mean absolute percentage error averages gaps relative to actuals, and mean absolute scaled error scales against a naive one step repeat:
MAE treats misses linearly and resists outliers. RMSE squares first, so one large miss dominates. MAPE speaks in percent but divides by , so breaks it and over forecasts score differently than under forecasts. MASE divides by the naive repeat forecast, so above means a simple repeat wins. That ladder is why backtesting reports more than one number.
Worked example: misses of 2 and 10
Take two points with absolute errors and . MAE is:
Squared errors are and . Mean square is . RMSE is:
The looks five times the under MAE since . It looks twenty five times under squaring since . That five against twenty five is the diagram headline: two equal bars on the left, one pair growing linearly in the middle, one pair exploding on the right.
What percent and scaled add
MAPE on missing by reads percent, which planners quote directly. On the same divides by zero and the formula fails, so teams clip, skip zeros, or switch metric. MASE answers a harder question: with MAE against naive MAE , MASE is , below , so the model beats doing nothing. Above , ship the naive repeat from exponential smoothing instead.
Code
Rebuild the worked example with plain Python and confirm and .
import math
errors: list[float] = [2.0, 10.0]mae: float = sum(abs(e) for e in errors) / len(errors)mse: float = sum(e * e for e in errors) / len(errors)rmse: float = math.sqrt(mse)
print(f"{mae:.1f} {rmse:.2f}") # -> 6.0 7.21print(f"{10.0 / 2.0:.0f} {(10.0**2) / (2.0**2):.0f}") # -> 5 25
actual: list[float] = [100.0, 150.0, 200.0, 50.0]forecast: list[float] = [110.0, 140.0, 180.0, 90.0]mae4: float = sum(abs(f - a) for a, f in zip(actual, forecast)) / len(actual)print(f"{mae4:.1f}") # -> 20.0The printout matches the hand arithmetic. Change the to and watch RMSE jump far faster than MAE, which is the square penalty in numbers.
Watch Out For
Quoting MAPE with zeros in the actuals
Percent error divides by actual, so one zero row makes the average infinite or silently huge.
Symptom: MAPE reads thousands of percent while MAE and RMSE look calm. Fix: filter or clip zeros before scoring, or report MASE from this page alongside backtesting windows that show where zeros cluster.
Tuning to RMSE while the business pays MAE
RMSE pushes the model to avoid rare large misses at the cost of slightly larger everyday misses. When stock cost is linear, that tradeoff loses money.
Symptom: RMSE improves release over release while holding cost or MAE stalls. Fix: pick the ruler from the cost first, as in classification metrics for classifiers, and keep the other metric as a diagnostic only.
The Quick Version
- Forecast evaluation picks the ruler that matches the business cost of being wrong.
- Errors 2 and 10 average to MAE 6.0 in our example.
- The same errors square to 4 and 100, mean 52, and root to RMSE 7.21.
- A miss of 10 reads 5 times a miss of 2 under MAE and 25 times under squaring.
- MAPE speaks in percent but fails when any actual equals zero.
- MASE below 1.0 proves the model beats a naive repeat, while above 1.0 favors the naive rule.