Gradient Boosting for Forecasting

Why Does This Exist?

Classical time series models like ARIMA have two major flaws:

1. They are strictly univariate. If you want to predict umbrella sales, ARIMA only looks at past umbrella sales. It cannot easily look at tomorrow's weather forecast.
2. They assume linear relationships.

Gradient Boosting models (like XGBoost or LightGBM) are the absolute kings of tabular data. They easily handle hundreds of external features (weather, holidays, price changes) and non-linear interactions. However, they don't natively understand "time". To use them for forecasting, we have to trick them by mathematically unrolling the timeline into a flat table.

Think of It Like This

How It Actually Works

Because XGBoost expects a flat table with rows (X) predicting a target (Y), you must perform Feature Engineering on your time series to convert it into this format.

1. Lag Features

If today is $T$, and you want to predict $T+1$, you create columns for what happened at $T-1$, $T-2$, etc.

• sales_lag_1: Sales yesterday.
• sales_lag_7: Sales exactly one week ago (crucial for capturing weekly seasonality).

2. Rolling Window Features

Instead of just single days, you provide aggregate summaries of the recent past.

• sales_rolling_mean_7d: The average sales over the last 7 days.
• sales_rolling_max_30d: The highest single day of sales in the last month.

3. Calendar Features

XGBoost doesn't know that December 25th is Christmas unless you tell it. You extract integers from the timestamp:

• day_of_week: (0 to 6).
• month: (1 to 12).
• is_holiday: (0 or 1).

Once this massive table is built, you train a standard Gradient Boosting Regressor to predict the target column based on the engineered feature columns.

Show Me the Code

In Python, you use pandas to engineer the lags, and then feed the resulting table into xgboost.

1import pandas as pd2import xgboost as xgb3
4# 1. Start with a simple time series5# Assume 'sales' contains daily sales data6df = pd.DataFrame({'sales': [10, 15, 12, 20, 25, 30, 22]})7
8# 2. Engineer Lag Features9df['lag_1'] = df['sales'].shift(1)10df['lag_2'] = df['sales'].shift(2)11
12# 3. Engineer Rolling Features13df['rolling_mean_3'] = df['sales'].shift(1).rolling(window=3).mean()14
15# Drop rows with NaN values created by shifting16df = df.dropna()17
18print("Engineered Tabular Data:")19print(df)20#    sales  lag_1  lag_2  rolling_mean_321# 3     20   12.0   15.0       12.33333322# 4     25   20.0   12.0       15.66666723# 5     30   25.0   20.0       19.00000024# 6     22   30.0   25.0       25.00000025
26# 4. Train XGBoost27X = df[['lag_1', 'lag_2', 'rolling_mean_3']]28y = df['sales']29
30model = xgb.XGBRegressor()31model.fit(X, y)32
33print("\nModel trained successfully!")

Watch Out For

The Quick Version

• Gradient boosting models (XGBoost, LightGBM) are powerful forecasters because they can ingest hundreds of external features (weather, price, holidays).
• They do not natively understand time.
• You must use Feature Engineering to convert chronological time into flat tabular columns (Lags, Rolling Averages, Calendar features).
• Tree-based models cannot extrapolate trends. You must de-trend the data first if it is growing indefinitely.

What to Read Next

• Deep Forecasting Models