Core ML

Stacking

Averaging models with fixed weights can't know which one to trust when. Train a meta-model on their predictions instead, and it learns the weighting for you.

Three diverse base models each hand their prediction to a meta-model, which learns from out-of-fold data how much to trust each one rather than averaging them with fixed weights

Why Does This Exist?

Bagging averages copies of the same kind of model. Boosting chains weak versions of the same kind of model. Neither has anything to say about combining models that are genuinely different in kind.

Here's the case we'll carry down the page. You're predicting next-day electricity demand for a grid operator: 50,000 past half-hours, weather, calendar, and yesterday's load as features. A gradient-boosted tree model nails the routine days and misses the cold snaps, where demand jumps in a way trees underpredict. A linear model with temperature-squared terms gets the cold snaps roughly right and is mediocre everywhere else. Ridge regression on twenty engineered lag features is worse than both alone but occasionally right when they're both wrong.

Averaging the three equally is arithmetic, not learning: it can't know the tree usually wins and the linear model only earns its keep below freezing. What you actually want is a model that has learned when to trust which one — and that means training something on their outputs.

Think of It Like This

A tumour board, not a majority vote

A hospital's tumour board doesn't average three specialists' opinions. A radiologist reads the scan, a pathologist reads the biopsy, an oncologist reads the patient's history — three different instruments, pointed at three different kinds of evidence, and each one is wrong in a different way on a different subset of cases.

Nobody averages "probably benign", "probably malignant", and "uncertain" into a blended verdict. A senior clinician learns, case by case and over years, that the pathologist is nearly always right on tissue calls and the radiologist is more useful for judging spread — and weighs their inputs by the kind of case in front of them, not by a fixed formula.

That weighing, learned from outcomes rather than fixed in advance, is what a stack's meta-model does with three algorithms' predictions instead of three specialists' opinions.

How It Actually Works

Two layers, and a name for each

A stack has base models — the tree, the linear model, the ridge fit, trained the ordinary way on the original features — and one meta-model, trained on a new dataset built entirely from the base models' predictions. For each half-hour, the meta-model's input isn't temperature and calendar day; it's three numbers, one prediction per base model. Its job is to learn the weighting the tumour board does by hand: when the tree's number can be trusted outright, and when it needs correcting toward the linear model's.

The meta-model is usually simple on purpose — a plain linear or logistic regression over the base predictions. It doesn't need capacity, because the hard modelling was already done downstream; it needs to learn a small number of blending weights without overfitting a dataset that has only as many rows as you had folds times base models.

The leak that decides whether stacking works at all

Here's the part that makes stacking either genuinely useful or a quiet way to overfit. Feed the meta-model predictions each base model made on rows it was trained on, and a model prone to memorising — a deep tree, a nearest-neighbour rule — will have near-perfect predictions there. The meta-model learns to lean on whichever base model memorised hardest, which is a real pattern in that data and complete noise on any other data.

The fix is the same one cross-validation uses to get an honest score: out-of-fold predictions. Split the training rows into $K$ folds. For each fold, fit every base model on the other $K-1$ folds and predict on the held-out one. Stack those held-out predictions back together and you get, for every training row, a base-model prediction from a model that never saw that row — exactly like a validation score, except kept as a feature instead of averaged into one number. Only predictions made this way go into the meta-model's training set.

Diversity is the only thing paying for the extra layer

Stack the same gradient-boosted tree trained three times with different random seeds and the meta-model has nothing to learn: three near-identical opinions correlate so strongly that no combination beats the best one alone, and you've spent three times the compute and inference cost for it. The models that fixed different parts of the demand curve — different assumptions, different features, different failure modes — are what a meta-model can actually exploit. Diminishing returns set in fast past three to five well-chosen base models; a tenth similar tree buys almost nothing.

Show Me the Code

Three base learners, one honest by construction — a memoriser that gets every training row right by finding itself as its own nearest neighbour. Stack it two ways.

1import numpy as np2
3rng = np.random.default_rng(7)4x = rng.normal(size=(400, 5))5y = 2.0 * x[:, 0] - x[:, 1] + rng.normal(0.0, 1.0, 400)6x_tr, x_te, y_tr, y_te = x[:300], x[300:], y[:300], y[300:]7
8def ridge(x_fit: np.ndarray, y_fit: np.ndarray, x_q: np.ndarray) -> np.ndarray:9    xb, xqb = np.c_[np.ones(len(x_fit)), x_fit], np.c_[np.ones(len(x_q)), x_q]10    w = np.linalg.solve(xb.T @ xb + np.eye(xb.shape[1]), xb.T @ y_fit)11    return xqb @ w12
13def memorizer(x_fit: np.ndarray, y_fit: np.ndarray, x_q: np.ndarray) -> np.ndarray:14    return np.array([y_fit[np.argmin(((x_fit - q) ** 2).sum(1))] for q in x_q])15
16def stack(use_folds: bool) -> float:17    ridge_p, mem_p = np.zeros(300), np.zeros(300)18    slices = [np.arange(300) % 5 != k for k in range(5)] if use_folds else [np.full(300, True)]19    for keep in slices:20        rows = ~keep if use_folds else keep21        ridge_p[rows] = ridge(x_tr[keep], y_tr[keep], x_tr[rows])22        mem_p[rows] = memorizer(x_tr[keep], y_tr[keep], x_tr[rows])23    meta_w = np.linalg.lstsq(np.c_[np.ones(300), ridge_p, mem_p], y_tr, rcond=None)[0]24    test_feats = np.c_[np.ones(100), ridge(x_tr, y_tr, x_te), memorizer(x_tr, y_tr, x_te)]25    return float(np.sqrt(np.mean((test_feats @ meta_w - y_te) ** 2)))26
27print(f"test RMSE: same-rows stack {stack(False):.2f} | out-of-fold stack {stack(True):.2f} | ridge alone {float(np.sqrt(np.mean((ridge(x_tr, y_tr, x_te) - y_te) ** 2))):.2f}")28# -> test RMSE: same-rows stack 1.86 | out-of-fold stack 1.05 | ridge alone 1.03

Trained on same-row predictions, the meta-model learns to trust the memoriser, which has memorised nothing useful about the test set: RMSE 1.86, worse than ridge alone. Trained on out-of-fold predictions, it correctly learns to ignore the memoriser and RMSE lands beside ridge's own 1.03 — no worse, and it would have improved had the memoriser been a real, diverse model instead of a strawman.

Watch Out For

Feeding the meta-model in-sample predictions

The single way stacking goes wrong. Symptom: a leaderboard-topping cross-validation score that collapses on the actual held-out test set, and a meta-model that puts almost all its weight on whichever base model overfits hardest — usually the most flexible one. The fix is always the same: every prediction that becomes a meta-model input must come from a base model that never trained on that row, via out-of-fold predictions in training and predictions from the fully-trained base models at inference time.

Stacking models that all make the same mistakes

Three gradient-boosted tree variants with slightly different hyperparameters correlate closely enough that a meta-model has almost nothing left to blend — you paid for three training runs and three inference calls to get one model's answer back. Diversity of kind is what a stack monetises: a tree family, a linear family, and something distance-based cover different failure modes; three trees cover one. Check pairwise correlation of the base predictions before adding a fourth model, not after.

The Quick Version

Stacking trains a meta-model on the predictions of several diverse base models, rather than voting or averaging them with fixed weights.
The meta-model is usually simple — a linear or logistic regression — because the hard modelling already happened one layer down.
Base predictions fed to the meta-model must be out-of-fold: each row's prediction from a model that never trained on that row.
Skip the out-of-fold step and the meta-model learns to trust whichever base model memorised training data hardest, and that flatters cross-validation while doing nothing for the real test set.
Diversity of model family, not just of hyperparameters, is what a meta-model can actually exploit; near-identical base models buy almost nothing.
Returns diminish fast. Three to five well-chosen base models capture most of the gain; a tenth similar one rarely helps.