Skip to content
AI360XpertBeta
Core ML

Visual explainer

Ensemble Stacking

Train a combiner on out-of-fold predictions, then apply it to refitted base models. A four-row fixture makes its learned weights checkable.

Equal weighting need not match the target.
Equal weighting need not match the target.

Stacking trains a model to combine other models' predictions. Suppose a tree predicts 5.2 and a linear model predicts 3.1 response units for a new query. Their equal average is 4.15, the diamond. A separately supplied true response of 3.5, the larger ring, makes that average's absolute error 0.65. The number line preserves these distances. We will return to this query after fitting a combiner on different rows; its target must not select the weights.

Exclude the Predicted Row

A held-out row must be excluded from each base fit.
A held-out row must be excluded from each base fit.

For row a, train each base model on b, c and d, then predict a. Its target is absent from those base fits. The diagram isolates fit membership: the dashed box marks the held-out row, and the bottom pair is its resulting tree/line prediction. Rotate this leave-one-row-out procedure across all four rows to assemble out-of-fold (OOF) features. The numbers used below are a supplied synthetic OOF matrix for explaining the meta-fit, not measured outputs from an executed tree and linear-model training run. Four singleton folds keep the example small; they are not a default cross-validation prescription.

Assemble the Meta-Data

Base predictions become columns in a new training matrix.
Base predictions become columns in a new training matrix.

Rows a–d have OOF prediction pairs (1,3)(1,3), (3,1)(3,1), (2,4)(2,4) and (4,2)(4,2). Their targets, separated to the right, are 2.5,1.5,3.5,2.52.5,1.5,3.5,2.5. Each row of the new feature matrix contains predictions from base fits that excluded that row. The target supervises the combiner. In this example the combiner receives only these two prediction columns, with no original input features. This distinction matters for the final failure scene.

Learn One Global Weight

One global weight minimizes this constrained blend's training loss.
One global weight minimizes this constrained blend's training loss.

Choose a deliberately restricted meta-model: g(T,L)=wT+(1−w)Lg(T,L)=wT+(1-w)L, with 0≤w≤10\leq w\leq1 and no intercept. Here TT and LL are tree and linear predictions. Minimize the sum of squared errors (SSE) over the four meta-training rows. Writing di=Ti−Lid_i=T_i-L_i gives d=(−2,2,−2,2)d=(-2,2,-2,2) and

w=∑idi(yi−Li)∑idi2=416=0.25.w=\frac{\sum_i d_i(y_i-L_i)}{\sum_i d_i^2}=\frac{4}{16}=0.25.

The line weight is therefore 0.75. The entire plotted loss is SSE(w)=16(w−0.25)2\mathrm{SSE}(w)=16(w-0.25)^2: zero at 0.25, one at 0.5, and nine at 1. Its geometry uses a common horizontal weight scale and vertical squared-response-unit scale. These are global coefficients; they do not switch with weather or region. More general meta-models may behave differently, and an unconstrained linear model need not have positive weights summing to one.

Check the Training Residuals

The fitted blend removes error on these meta-training rows.
The fitted blend removes error on these meta-training rows.

The equal average predicts (2,2,3,3)(2,2,3,3). Prediction minus target gives (−0.5,+0.5,−0.5,+0.5)(-0.5,+0.5,-0.5,+0.5), shown on the upper signed-error strip. The fitted blend predicts (2.5,1.5,3.5,2.5)(2.5,1.5,3.5,2.5), with four zero residuals below. Equal averaging has SSE 1 and MSE 0.25; the fitted blend has zero meta-training error. Although its inputs are OOF base predictions, these same four rows trained the meta-model. They cannot provide an unbiased evaluation of the full stack. Use a separate test set or an outer validation procedure that refits the entire stack.

Refit the Base Layer

Prediction uses base models refitted on the full training set.
Prediction uses base models refitted on the full training set.

After collecting OOF features, refit the base models on all available training rows a–d for deployment. The meta-model keeps the coefficients learned from OOF features. The diagram shows this preparation stage; it does not show a new query being fitted. Refit models can produce a different prediction distribution from the smaller fold models. More folds, different base models or a more flexible combiner increase fitting cost and can change generalization. A linear meta-model can also overfit; no sample-count threshold guarantees that it is safe.

Avoid Memorized Evidence

Training predictions can hide errors that OOF predictions expose.
Training predictions can hide errors that OOF predictions expose.

For row a, the true response is 2.5, marked by a diamond. The upper lane is a hypothetical memorizing base model predicting exactly 2.5 after seeing a during fitting. Its training error reveals none of the mistakes a combiner needs to learn about. The lower lane shows our supplied OOF values: tree 1, square, and line 3, circle. Their distances from 2.5 are real nonzero errors. Both lanes share a response scale. Folds also need to respect groups or time, and learned preprocessing must stay inside each fold. OOF construction does not guarantee freedom from every kind of leakage.

Predict Without the Target

The learned weights combine a new pair into 3.625.
The learned weights combine a new pair into 3.625.

Return to the separate query from scene 1. Suppose the refitted tree and line output 5.2 and 3.1. The frozen combiner computes 0.25(5.2)+0.75(3.1)=1.3+2.325=3.6250.25(5.2)+0.75(3.1)=1.3+2.325=3.625. Each contribution bar uses 100 pixels per response unit; the bottom bar concatenates the same two lengths. The full prediction can be made without knowing the query's target. This scene assumes the base outputs; it does not claim the supplied OOF matrix determines what refitted base models must predict on unseen inputs.

Evaluate the Frozen Stack

A separate target checks the frozen stack on a new case.
A separate target checks the frozen stack on a new case.

Now compare with the query's independently supplied target 3.5. The stack's absolute error is ∣3.625−3.5∣=0.125|3.625-3.5|=0.125, versus the equal average's 0.65. Both gaps use the same response scale as scene 1, with vertical offsets to keep the marks distinct. This is one illustrative query, not a measured average improvement, and it must remain outside weight selection. Selecting among many stacks on the same query would turn it into tuning data.

Where It Breaks

Identical prediction features cannot distinguish different targets.
Identical prediction features cannot distinguish different targets.

Consider a separate failure fixture: two cases produce exactly the same prediction tuple (2,4)(2,4) but have targets 3 and 5. A deterministic combiner receiving only that tuple must give them the same output. Even if we fit a new combiner specifically to these cases, the best common squared-loss prediction is 4: (4−3)2+(4−5)2=2(4-3)^2+(4-5)^2=2. The two crosses show unavoidable misses around that shared value. This newly optimized value is not the earlier frozen blend, which would output 3.5 for (2,4)(2,4). Original features might distinguish the cases, but this prediction-only stack discarded that information. Identical base models do not categorically prevent correction; the precise limit is indistinguishable meta-inputs with different targets.

The Quick Version

  • OOF predictions become the meta-model's training features.
  • The predicted row stays outside each base fit.
  • A constrained four-row blend learns weights 0.25 and 0.75.
  • Linear blend weights are global in this example.
  • OOF meta-training error is not full-stack test error.
  • Refit base models on all training data for inference.
  • Frozen weights predict without the unseen target.
  • Identical meta-inputs cannot reveal different targets.

Mechanism reference: scikit-learn StackingRegressor documentation. The matrix, fit and query checks are synthetic calculations, not performance benchmarks.