Visual explainer
Ensemble Stacking
Train a combiner on out-of-fold predictions, then apply it to refitted base models. A four-row fixture makes its learned weights checkable.
Stacking trains a model to combine other models' predictions. Suppose a tree predicts 5.2 and a linear model predicts 3.1 response units for a new query. Their equal average is 4.15, the diamond. A separately supplied true response of 3.5, the larger ring, makes that average's absolute error 0.65. The number line preserves these distances. We will return to this query after fitting a combiner on different rows; its target must not select the weights.
Exclude the Predicted Row
For row a, train each base model on b, c and d, then predict a. Its target is absent from those base fits. The diagram isolates fit membership: the dashed box marks the held-out row, and the bottom pair is its resulting tree/line prediction. Rotate this leave-one-row-out procedure across all four rows to assemble out-of-fold (OOF) features. The numbers used below are a supplied synthetic OOF matrix for explaining the meta-fit, not measured outputs from an executed tree and linear-model training run. Four singleton folds keep the example small; they are not a default cross-validation prescription.
Assemble the Meta-Data
Rows a–d have OOF prediction pairs , , and . Their targets, separated to the right, are . Each row of the new feature matrix contains predictions from base fits that excluded that row. The target supervises the combiner. In this example the combiner receives only these two prediction columns, with no original input features. This distinction matters for the final failure scene.
Learn One Global Weight
Choose a deliberately restricted meta-model: , with and no intercept. Here and are tree and linear predictions. Minimize the sum of squared errors (SSE) over the four meta-training rows. Writing gives and
The line weight is therefore 0.75. The entire plotted loss is : zero at 0.25, one at 0.5, and nine at 1. Its geometry uses a common horizontal weight scale and vertical squared-response-unit scale. These are global coefficients; they do not switch with weather or region. More general meta-models may behave differently, and an unconstrained linear model need not have positive weights summing to one.
Check the Training Residuals
The equal average predicts . Prediction minus target gives , shown on the upper signed-error strip. The fitted blend predicts , with four zero residuals below. Equal averaging has SSE 1 and MSE 0.25; the fitted blend has zero meta-training error. Although its inputs are OOF base predictions, these same four rows trained the meta-model. They cannot provide an unbiased evaluation of the full stack. Use a separate test set or an outer validation procedure that refits the entire stack.
Refit the Base Layer
After collecting OOF features, refit the base models on all available training rows a–d for deployment. The meta-model keeps the coefficients learned from OOF features. The diagram shows this preparation stage; it does not show a new query being fitted. Refit models can produce a different prediction distribution from the smaller fold models. More folds, different base models or a more flexible combiner increase fitting cost and can change generalization. A linear meta-model can also overfit; no sample-count threshold guarantees that it is safe.
Avoid Memorized Evidence
For row a, the true response is 2.5, marked by a diamond. The upper lane is a hypothetical memorizing base model predicting exactly 2.5 after seeing a during fitting. Its training error reveals none of the mistakes a combiner needs to learn about. The lower lane shows our supplied OOF values: tree 1, square, and line 3, circle. Their distances from 2.5 are real nonzero errors. Both lanes share a response scale. Folds also need to respect groups or time, and learned preprocessing must stay inside each fold. OOF construction does not guarantee freedom from every kind of leakage.
Predict Without the Target
Return to the separate query from scene 1. Suppose the refitted tree and line output 5.2 and 3.1. The frozen combiner computes . Each contribution bar uses 100 pixels per response unit; the bottom bar concatenates the same two lengths. The full prediction can be made without knowing the query's target. This scene assumes the base outputs; it does not claim the supplied OOF matrix determines what refitted base models must predict on unseen inputs.
Evaluate the Frozen Stack
Now compare with the query's independently supplied target 3.5. The stack's absolute error is , versus the equal average's 0.65. Both gaps use the same response scale as scene 1, with vertical offsets to keep the marks distinct. This is one illustrative query, not a measured average improvement, and it must remain outside weight selection. Selecting among many stacks on the same query would turn it into tuning data.
Where It Breaks
Consider a separate failure fixture: two cases produce exactly the same prediction tuple but have targets 3 and 5. A deterministic combiner receiving only that tuple must give them the same output. Even if we fit a new combiner specifically to these cases, the best common squared-loss prediction is 4: . The two crosses show unavoidable misses around that shared value. This newly optimized value is not the earlier frozen blend, which would output 3.5 for . Original features might distinguish the cases, but this prediction-only stack discarded that information. Identical base models do not categorically prevent correction; the precise limit is indistinguishable meta-inputs with different targets.
The Quick Version
- OOF predictions become the meta-model's training features.
- The predicted row stays outside each base fit.
- A constrained four-row blend learns weights 0.25 and 0.75.
- Linear blend weights are global in this example.
- OOF meta-training error is not full-stack test error.
- Refit base models on all training data for inference.
- Frozen weights predict without the unseen target.
- Identical meta-inputs cannot reveal different targets.
Mechanism reference: scikit-learn StackingRegressor documentation. The matrix, fit and query checks are synthetic calculations, not performance benchmarks.