Skip to content
AI360Xpert
Visual Explainers
Visual explainer

Bagging vs Boosting

Two ensemble strategies, two different problems: bagging trains parallel trees on random samples to squash variance; boosting chains sequential trees to fix bias one residual at a time.

Two ensemble strategies exist: bagging averages parallel trees to cut variance; boosting chains sequential trees to cut bias.
Two ensemble strategies exist: bagging averages parallel trees to cut variance; boosting chains sequential trees to cut bias.

Both strategies start from the same premise — one tree is not enough — but they diagnose the problem differently. Bagging says the tree is too sensitive to which data points it happened to see. Boosting says the tree is too simple to see the pattern at all.

Bagging: Parallel Bootstrap

Bagging samples the data with replacement to create diverse training sets, then trains all trees in parallel.
Bagging samples the data with replacement to create diverse training sets, then trains all trees in parallel.

Each tree draws a fresh random sample — with replacement — from the full dataset. Trees disagree because they saw different slices. All trees train simultaneously; no tree waits for another. Their outputs are averaged or majority-voted at the end.

Boosting: Sequential Weighting

Boosting trains each tree sequentially, weighting the samples that the previous tree got wrong.
Boosting trains each tree sequentially, weighting the samples that the previous tree got wrong.

After each learner, the samples it misclassified are given higher weight. The next learner must focus on those hard cases. The sequence can only proceed one step at a time — no parallelism. Each tree is deliberately kept shallow, a weak learner alone, strong in the chain.

Different Problems, Different Fixes

Bagging reduces variance by averaging; boosting reduces bias by targeting the systematic error.
Bagging reduces variance by averaging; boosting reduces bias by targeting the systematic error.

Averaging many uncorrelated predictions reduces the variance of the combined output. Targeting residuals corrects systematic mistakes that a single tree is too weak to model. These are distinct operations acting on different components of the error.

Where It Breaks

Noisy labels break boosting by forcing it to memorise mislabelled points; bagging merely degrades gracefully.
Noisy labels break boosting by forcing it to memorise mislabelled points; bagging merely degrades gracefully.

Boosting up-weights the examples its current ensemble gets wrong. A mislabelled example is always wrong — so it gets up-weighted every round until the model bends its decision boundary to reach it. Bagging averages across many trees, so a noisy point influences only the trees whose sample happened to include it, and the damage stays local.

The Quick Version

  • Bagging: parallel, independent trees; each sees a random data bootstrap.
  • Boosting: sequential trees; each corrects the previous one's mistakes.
  • Bagging reduces variance; boosting reduces bias.
  • Both need weak base learners — strong base learners kill the ensemble benefit.
  • Noisy labels are safe for bagging; they poison boosting systematically.

Related Concepts