Skip to content
AI360XpertBeta
Core ML

Visual explainer

Ensemble Methods

Combine predictions by voting, averaging, sequential correction or a trained combiner. Different mistakes can help; shared mistakes survive.

One model can miss a row that other models recover.
One model can miss a row that other models recover.

An ensemble combines several predictions into one. Its value depends on what the members get wrong and how their outputs combine. Start with four held-out rows T1–T4 whose true classes are (1,1,0,0)(1,1,0,0). Model A predicts (0,1,0,0)(0,1,0,0), missing T1. The ring marks that mismatch. This is a supplied synthetic prediction fixture, not a benchmark or a claim that we trained a particular tree to produce these values.

Change the Training View

Resampling changes which training rows each member sees.
Resampling changes which training rows each member sees.

Here lowercase a–d identify a separate training set, not the held-out T rows. Drawing four rows with replacement gives bags aabd and bccd: the dashed outlines mark repeated rows. Bagging uses such randomized training views to produce multiple fitted members. Sampling can change their errors, but it does not guarantee different mistakes. Other ways to create diversity include feature subsets and different model families. This drawing explains the sampling operation; it does not assert these two bags generated our supplied classifier outputs.

Count the Votes

Two votes can overrule one wrong prediction.
Two votes can overrule one wrong prediction.

For held-out T1, models A, B and C predict 0, 1 and 1. The bar counts one vote for class 0 and two for class 1; its segment lengths have a 1:2 ratio. Hard voting chooses class 1 and repairs A's error on this row. That result says nothing yet about another row or about the probability that class 1 is correct.

Labels or Probabilities?

Hard voting and soft voting can disagree.
Hard voting and soft voting can disagree.

Switch to a separate query with model probabilities for class 1 of (0.51,0.51,0.01)(0.51,0.51,0.01). Thresholding each at 0.5 gives hard labels (1,1,0)(1,1,0) and a hard-vote result of 1. Soft voting instead averages probabilities: (0.51+0.51+0.01)/3≈0.343(0.51+0.51+0.01)/3\approx0.343. The diamond marks that mean below the dashed 0.5 threshold, giving class 0. Bar lengths use the same probability scale. No true label is given for this query, so neither aggregation rule is declared better here. Probability quality and calibration matter when selecting soft voting.

Inspect Every Error

Different error locations let this majority recover all four rows.
Different error locations let this majority recover all four rows.

Return to T1–T4. B predicts (1,0,0,0)(1,0,0,0) and C predicts (1,1,1,0)(1,1,1,0). Each model misses one different row, circled in the matrix. On every column, at least two models match the truth. The majority is (1,1,0,0)(1,1,0,0): each member has accuracy 3/43/4, while the majority has 4/44/4 on this tiny fixture. This directly counted example replaces an unsupported general accuracy claim. It illustrates a possibility, not a population estimate. Even distinct algorithms can share errors on fresh data.

Average Numerical Outputs

Regression averages numerical predictions rather than class labels.
Regression averages numerical predictions rather than class labels.

For a separate regression query, suppose the true response is 4 and three members predict 2, 4 and 6 response units. Their average is (2+4+6)/3=4(2+4+6)/3=4, shown by the diamond. Signed errors −2,0,+2-2,0,+2 cancel. Averaging is an operation on numerical outputs, unlike taking the most frequent class label. The truth is supplied to evaluate the result, not supplied to the ensemble at prediction time.

Shared Errors Persist

Shared signed errors survive averaging.
Shared signed errors survive averaging.

Both lanes share a signed-error scale; circles identify member errors and diamonds their average. Two errors (−2,+2)(-2,+2) average to zero. Two errors (+2,+2)(+2,+2) average to +2+2; their circles are offset vertically so both remain visible. Duplicating a prediction adds no cancellation. This is an exact statement about these errors. Estimating variance and error correlation across a data distribution needs more than one query.

Build Corrections in Order

Boosting constructs a sum through sequential corrections.
Boosting constructs a sum through sequential corrections.

A separate scalar squared-loss example has target 4 and current prediction 2. If each new learner fits the current residual exactly and contributes half of it, the predictions move 2→3→3.52\rightarrow3\rightarrow3.5. Arrow lengths show contributions 1 and 0.5 on one scale; dashed gaps show residuals 2, 1 and 0.5. This local example isolates boosting's dependency on the ensemble already built. Real trees fit many rows jointly, so they cannot generally fit every residual exactly. See gradient boosting for a full tree fixture.

Learn the Combination

Stacking trains a combiner on held-out base predictions.
Stacking trains a combiner on held-out base predictions.

Stacking uses member predictions as features for another model. For T1, the illustrated feature tuple is (0,1,1)(0,1,1) and the supervised target is 1. OOF means out-of-fold: each base prediction must come from a fit that excluded T1. The target supervises the meta-model, shown entering by a separate path. This single row cannot determine a fitted combiner; the drawing deliberately shows no claimed meta-model output. Ensemble stacking follows a complete training and inference example.

Where It Breaks

A unanimous vote can still be wrong.
A unanimous vote can still be wrong.

On a new query, suppose all three models predict 1 while the true class is 0. The majority is wrong even though every member agrees. The cross compares the vote with the held-out truth. A vote fraction is not automatically calibrated confidence, and model-family diversity does not guarantee independent errors. Ensembles add fitting, prediction and maintenance costs; retain them only when evaluation on properly held-out data justifies those costs.

The Quick Version

  • Hard voting combines class labels by counts.
  • Soft voting combines probabilities before choosing a class.
  • Regression averaging can cancel opposite signed errors.
  • Shared mistakes survive aggregation.
  • Bagging randomizes training views.
  • Boosting adds dependent corrections.
  • Stacking learns from out-of-fold prediction features.
  • Unanimity does not establish correctness.

Mechanism reference: scikit-learn ensemble guide. The displayed prediction fixtures are illustrative, not measured model performance.