Backtesting (Time Series)
Backtesting replays the past the way it actually unfolded: train only on what came before, predict what came next, and average the misses.
Why Does This Exist?
The default way to grade a model is to shuffle the rows, hold some back, and score. That works when rows are independent. It breaks the moment order is the data. Say you forecast Friday's price and your shuffled fold trained on Monday, Tuesday, and Saturday. Saturday already knows how Friday turned out, so your model is not forecasting. It is remembering, and the score lies to you. That leak has a name, data leakage, and on ordered data it is the easiest mistake to make with total confidence.
Backtesting is the chronological version of that grading ritual. You walk through history the way production walks through time: stand at some week, train on everything before it, predict the next point, write down the miss, step forward, repeat. The average miss is your honest estimate of deployment error.
This page covers the replay discipline itself, with the numbers worked by hand. The fold mechanics get their full treatment in time series cross-validation. Choosing the miss you average, MAE or RMSE or MAPE, belongs to forecast evaluation. The short glossary form lives in the backtesting definition.
Think of It Like This
The student who hides the answer key
A student claims she can solve any past exam. You test her with the 2022 paper, but she revised from worked solutions that include 2022. Of course she nails it. She has seen the answers, not learned the subject. The honest test is stricter: she may study everything up to 2021, then sits 2022 cold. Then she may study 2022 and sits 2023 cold. Her average over those cold sittings is what her next unseen exam will look like.
Each cold sitting is one backtest fold. The covered answers are the rule that training must end before the test point begins. The average is the deployment estimate.
Where it stops: exams reuse one syllabus, so old papers stay relevant. Real series drift, regimes change, and last year's answers can go stale. That is why the rolling window exists, and why no replay score survives a series that changed its nature halfway through.
How It Actually Works
The one rule every fold obeys
Write the series , where the subscript is time and means the value at week . A fold picks a cutoff , trains only on through , and scores the forecast of , the next unseen point. The miss is , plain absolute distance. Average the misses over all folds and you get the mean absolute error, the MAE. One rule, no exceptions: the test point always sits strictly after every row that trained it. Which miss to average is a separate decision, and forecast evaluation settles it.
Shuffling violates this rule by construction. A shuffled fold tests with a model trained on through , so it interpolates between known neighbours instead of extrapolating into the unknown. Interpolation is the easier problem, which is exactly why the shuffled score reads lower and means less. The gap grows with autocorrelation: the more today resembles yesterday, the more peeking pays.
Expanding or rolling: how much history each fold keeps
The two windows agree on the rule and differ only in how much past each fold may use. An expanding window starts at every time and grows: fold one trains on weeks 1 to 6, fold two on 1 to 7, fold three on 1 to 8. It uses all available history, which matches how you will actually run the model, but refits get slower as history piles up. A rolling window keeps training length fixed and slides it forward, dropping old weeks as new ones arrive. It trains faster and forgets on purpose, which wins when the process drifts and last year's regime misleads this year's forecast.
Pick by stability. A stable process rewards expanding, because old rows still describe the world. A drifting one punishes it, because stale rows vote against the present. If you are unsure whether your series is stable enough for expanding, stationarity is the check to run first. Either way, the folds stay chronological. The window changes how much past you keep, never whether the future may leak in.
Worked example
Ten weeks of demand: . The forecasting rule is the naive one: predict that next week equals this week. Three expanding folds, one step each:
- Fold 1. Train on weeks 1 to 6, stand at , predict week 7. Forecast , actual , miss .
- Fold 2. Train on weeks 1 to 7, stand at , predict week 8. Forecast , actual , miss .
- Fold 3. Train on weeks 1 to 8, stand at , predict week 9. Forecast , actual , miss .
Average the three misses:
That 1.67 is the honest number: the average cold-sitting miss of the naive rule on this series. A shuffled fold on the same data would score lower by letting training surround each test point, and that lower number would describe nothing you can deploy.
Code
The loop below replays the worked example exactly: same ten weeks, same naive rule, same three expanding folds.
series: list[float] = [10.0, 12.0, 11.0, 13.0, 15.0, 14.0, 16.0, 18.0, 17.0, 19.0]
misses: list[float] = []for end in (6, 7, 8): forecast: float = series[end - 1] # naive rule: next week equals this week misses.append(abs(series[end] - forecast))
print(misses) # -> [2.0, 2.0, 1.0]
print(f"mean miss {sum(misses) / len(misses):.2f}") # -> mean miss 1.67The printout reads [2.0, 2.0, 1.0] and mean miss 1.67, matching the hand computation fold for fold. Notice the training slice always ends before the test index. That ordering is the entire method.
Watch Out For
Shuffling a series out of habit
KFold with shuffling is the snippet everyone has memorized, and it runs without complaint on ordered data. The scores come back low and stable, because every fold cheats the same way, and nothing in the output says the numbers are fiction.
Symptom: a suspiciously good error on data that has a date column, a timestamp, or any natural order, with no splitter chosen on purpose. Fix: check for ordering before you pick a splitter. If rows have a time axis, reach for a chronological splitter and never shuffle.
Normalizing with the whole series before splitting
Even with perfect chronological folds, you can still leak. Compute the mean over all ten weeks, subtract it everywhere, then split, and every training fold now carries a whisper of the test points inside that mean. Same story for any feature built from full-series statistics.
Symptom: a backtest that looks great and a deployed model that immediately disappoints, with no bug in the fold loop. Fix: fit every scaler, imputer, and engineered feature on the training portion of each fold only, then apply it to that fold's test point.
The Quick Version
- Backtesting grades a forecaster by replaying history forward: train on the past, predict the next point, average the misses.
- Every test point must sit strictly after every row that trained it, which is why shuffled cross-validation is invalid on ordered data.
- An expanding window grows training fold by fold, while a rolling window keeps it fixed and drops old rows that may describe a dead regime.
- A naive next-equals-now rule over three folds misses by 2, 2 and 1, for an honest MAE of 1.67.
- Fit scalers and features inside each training fold, because full-series preprocessing leaks the future just as surely as shuffling does.