Skip to content
AI360XpertBeta

Stacking Ensemble Learning

Averaging models with fixed weights can't know which one to trust when. Train a meta-model on their predictions instead, and it learns the weighting for you.

Three diverse base models feed out-of-fold predictions into a meta-model that learns how much to trust each one, producing a blended prediction of 2.03 from 2.1, 1.8 and 2.0
Three diverse base models feed out-of-fold predictions into a meta-model that learns how much to trust each one, producing a blended prediction of 2.03 from 2.1, 1.8 and 2.0

Why Does This Exist?

You predict next day electricity demand from 50,000 past half hours plus weather and calendar. A tree nails routine days and misses cold snaps. A linear model with a temperature squared term catches the snaps and trails elsewhere. Ridge on lag features loses to both alone yet is right on a few odd mornings.

A plain average cannot say the tree usually wins and the linear model only earns its keep below freezing. You want a weighting that depends on the models outputs, learned from outcomes. That is what a second layer trained on first layer predictions gives you.

One idea underneath it. Base models map features to predictions. A meta-model maps those predictions to one final number. Its training rows are not temperature and day of week. Each row is three numbers, one prediction per base model, plus the true demand.

This page covers the two layers, the leak that decides if it works, and why diversity pays. Same kind averaging lives in bagging. Sequential fixing of errors lives in boosting.

Think of It Like This

A tumour board, not a majority vote

A radiologist reads the scan, a pathologist reads the biopsy, an oncologist reads the history. Three different instruments, three different blind spots. The board does not average probably benign with probably malignant. A senior clinician learns over cases that the pathologist owns tissue calls while the radiologist owns spread, and weighs them by case type.

Base models are the three specialists. Their predictions are the three opinions. The meta-model is the learned weighing, fitted on past cases with known outcomes rather than fixed in advance.

Where it stops: a board cannot fix three identical opinions. Three copies of one scan reader add no view. Stacking only pays when base models fail on different rows, which is why three similar trees buy almost nothing.

How It Actually Works

Name the parts. Base models train the ordinary way on features. The meta-model trains on a new table built from base predictions. For each half hour its input is three predictions, and its target is true demand. In practice the meta-model stays simple, often linear or logistic over those few inputs, because the hard modelling already happened below and a flexible top layer would overfit a narrow table.

The leak decides everything. Train base models on all rows and predict those same rows, and a memorizer looks perfect there. A deep tree or nearest neighbour rule can quote training rows back. The meta-model then learns to trust the memorizer most, which is true in sample and false out of sample.

Out-of-fold predictions fix the leak

Split training rows into K folds. For each fold fit every base model on the other K minus 1 folds and predict the held out fold. Stack those held out predictions and every row has a prediction from a model that never saw it. That table trains the meta-model. At inference you use base models fitted on all training rows, then the frozen meta-model. The honest table in training matches the real use at inference.

Worked example with three numbers

Say true demand is 2.02.0 and base predictions are 2.12.1, 1.81.8 and 2.02.0. A learned meta-model uses weights 0.70.7, 0.20.2 and 0.10.1. The blend is 0.7×2.1+0.2×1.8+0.1×2.0=1.47+0.36+0.20=2.030.7 \times 2.1 + 0.2 \times 1.8 + 0.1 \times 2.0 = 1.47 + 0.36 + 0.20 = 2.03. Error is 0.030.03. Equal weights would give 1.971.97 with error 0.030.03 the other way, but the learned weights also encode that the first model usually deserves more say, which an average cannot know.

Diversity is what the top layer buys

Three gradient boosted trees with different seeds correlate near 0.990.99. No weighting beats the best one alone by much, and you paid three inference calls for it. A tree plus a linear model plus a distance rule correlate far less, so the meta-model can lean tree on routine days and linear below freezing. Past three to five well chosen families, gains flatten fast.

Code

Three base learners with one honest by construction memorizer. Same row stacking trusts the memorizer and pays on test. Out-of-fold stacking learns to ignore it.

import numpy as nprng = np.random.default_rng(7)x = rng.normal(size=(400, 5))y = 2.0 * x[:, 0] - x[:, 1] + rng.normal(0.0, 1.0, 400)x_tr, x_te, y_tr, y_te = x[:300], x[300:], y[:300], y[300:]def ridge(x_fit: np.ndarray, y_fit: np.ndarray, x_q: np.ndarray) -> np.ndarray:    xb = np.c_[np.ones(len(x_fit)), x_fit]; xqb = np.c_[np.ones(len(x_q)), x_q]    w = np.linalg.solve(xb.T @ xb + np.eye(xb.shape[1]), xb.T @ y_fit)    return xqb @ wdef memorizer(x_fit: np.ndarray, y_fit: np.ndarray, x_q: np.ndarray) -> np.ndarray:    return np.array([y_fit[int(np.argmin(((x_fit - q) ** 2).sum(1)))] for q in x_q])def stack_rmse(use_folds: bool) -> float:    rp = np.zeros(300); mp = np.zeros(300)    sl = [np.arange(300) % 5 != k for k in range(5)] if use_folds else [np.full(300, True)]    for keep in sl:        rows = ~keep if use_folds else keep        rp[rows] = ridge(x_tr[keep], y_tr[keep], x_tr[rows])        mp[rows] = memorizer(x_tr[keep], y_tr[keep], x_tr[rows])    mw = np.linalg.lstsq(np.c_[np.ones(300), rp, mp], y_tr, rcond=None)[0]    tf = np.c_[np.ones(100), ridge(x_tr, y_tr, x_te), memorizer(x_tr, y_tr, x_te)]    return float(np.sqrt(np.mean((tf @ mw - y_te) ** 2)))print(f"same {stack_rmse(False):.2f} oof {stack_rmse(True):.2f} ridge {float(np.sqrt(np.mean((ridge(x_tr, y_tr, x_te) - y_te) ** 2))):.2f}")# -> same 1.86 oof 1.05 ridge 1.03

Same row stacking lands at 1.861.86, worse than ridge alone at 1.031.03. Out-of-fold stacking lands at 1.051.05, beside ridge, because it learned the memorizer deserves no weight. With a truly diverse second model instead of a strawman memorizer, that same machinery improves on the best base alone.

Watch Out For

Feeding the meta-model in-sample predictions

Scores look great in validation and collapse on the held out test. The meta-model puts almost all weight on the most flexible base model, usually the one that overfits hardest.

Symptom: cross-validation topping the board plus test far worse, with meta weights concentrated on one complex base. Fix: rebuild the meta training table so every prediction comes from a model that never trained on that row, via out-of-fold predictions, and refit. At inference use fully fitted base models plus the frozen meta-model.

Stacking models that make the same mistakes

Three tree variants with different depths correlate so closely that blending cannot beat picking the best one. You paid for extra training and slower inference to get one answer back.

Symptom: pairwise correlation of base predictions above about 0.950.95 and meta weights unstable across folds. Fix: check correlation before adding a fourth model. Add a different family, a different feature view, or stop at one strong model. Diversity of kind is what a stack monetises.

The Quick Version

  • Stacking trains a meta-model on base model predictions rather than voting with fixed weights.
  • Our three predictions 2.1, 1.8 and 2.0 blend with weights 0.7, 0.2 and 0.1 to 2.03, encoding trust an average cannot know.
  • The meta-model stays simple because the hard modelling already happened one layer down.
  • Meta inputs must be out-of-fold, or the top layer learns to trust whichever base memorised hardest.
  • Same row stacking scored 1.86 against ridge alone at 1.03, while out-of-fold scored 1.05 by ignoring the memorizer.
  • Diversity of model family buys the gain, and returns flatten past three to five well chosen bases.