Core ML

Backtesting (Time Series)

Evaluating a forecasting model by simulating how it would have performed in the past, strictly using expanding or rolling windows to ensure the model never 'peeks' into the future.

Standard K-Fold Cross Validation randomly shuffles data. In time series, you must use Expanding or Rolling Windows to maintain strict chronological order.

Why Does This Exist?

In standard machine learning (like predicting if an image is a cat or a dog), we use K-Fold Cross Validation. We randomly shuffle the dataset, hide 20% of the images, train on 80%, and see how the model does.

You cannot randomly shuffle a time series.

If you are trying to predict the stock market on Friday, you cannot train your model on Monday, Tuesday, and Saturday. That is a catastrophic data leak. Your model is literally peeking into the future to predict the past.

Backtesting is the chronological equivalent of cross-validation. It guarantees that the model is evaluated exactly as it would have performed in the real world, moving forward through time without ever seeing tomorrow's data.

Think of It Like This

Imagine you claim you have a system that can predict the winning lottery numbers.

If you test your system by looking at all the winning numbers from 2020 to 2024, finding a pattern, and then "predicting" the numbers for 2022, nobody will believe you. You already knew the answer for 2022.

The only way to prove your system works is to sit down on January 1st, 2023. You predict January 2nd. We check if you were right. You then update your system using January 2nd's data to predict January 3rd. We check if you were right. Backtesting forces the model to step forward through history day by day, proving it can actually predict the unknown.

How It Actually Works

There are two primary ways to backtest a time series model.

1. Expanding Window (Walk-Forward)

This is the most common approach. The size of the training dataset grows larger at every step.

Fold 1: Train on Year 1. Predict Q1 of Year 2. Calculate error.
Fold 2: Train on Year 1 + Q1 of Year 2. Predict Q2 of Year 2. Calculate error.
Fold 3: Train on Year 1 + Q1 + Q2 of Year 2. Predict Q3...

Pros: The model uses all available historical data, which is exactly how you will use it in production.
Cons: The training data gets massive over time, which can slow down training significantly.

2. Rolling Window

The size of the training dataset stays fixed. As new data comes in, old data is thrown away.

Fold 1: Train on Jan $\rightarrow$ Dec. Predict January.
Fold 2: Train on Feb $\rightarrow$ Jan. Predict February.

Pros: Faster to train. Excellent for highly volatile markets (like crypto) where data from 3 years ago is actively harmful and irrelevant to today's market dynamics.

Show Me the Code

You rarely write the rolling window loops yourself. scikit-learn provides a dedicated splitter for time series data.

1import numpy as np2from sklearn.model_selection import TimeSeriesSplit3
4# Dummy daily data: 10 days5X = np.array([[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]])6y = np.array([10, 20, 30, 40, 50, 60, 70, 80, 90, 100])7
8# TimeSeriesSplit implements Expanding Window backtesting9# We ask for 3 splits10tscv = TimeSeriesSplit(n_splits=3)11
12fold = 113for train_index, test_index in tscv.split(X):14    print(f"--- Fold {fold} ---")15    print(f"TRAIN indices: {train_index}")16    print(f"TEST  indices: {test_index}\n")17    fold += 118
19# Output:20# --- Fold 1 ---21# TRAIN indices: [0 1 2 3]22# TEST  indices: [4 5]23#24# --- Fold 2 ---25# TRAIN indices: [0 1 2 3 4 5]26# TEST  indices: [6 7]27#28# --- Fold 3 ---29# TRAIN indices: [0 1 2 3 4 5 6 7]30# TEST  indices: [8 9]

Notice how the training set strictly grows, and the test set is always strictly after the training set.

Watch Out For

Lookahead Bias. Even with a proper TimeSeriesSplit, you can still leak data if you perform your Feature Engineering on the entire dataset before splitting. For example, if you calculate the overall mean of the dataset and subtract it from every row, your training data is now polluted with information from the test data. You must calculate scaling and normalization parameters strictly on the training folds.

The Quick Version

Standard randomized Cross-Validation is invalid for time series because it leaks future data into the past.
Backtesting simulates reality by forcing the model to move chronologically through time.
Expanding Window: The training set grows continuously as you step forward.
Rolling Window: The training set stays a fixed size, dropping old data as it absorbs new data.
Proper backtesting is the only way to prove a forecasting model will actually work in production.

Related concepts

Forecast Evaluation