Boosting

Why Does This Exist?

Some models aren't unstable. They're wrong in the same direction every time, and averaging does nothing at all for that.

Here's the case we'll carry down the page. A batching plant mixes concrete and you want the 28-day compressive strength in megapascals before the truck leaves. Eight numbers per batch: cement, water, fly ash, aggregate, superplasticiser, slag, curing days. A thousand past batches, one crushed cylinder each.

Fit a depth-2 tree to that. Four leaves, four predicted strengths, roughly 8 MPa off across most of the yard. Now fit five hundred of them the way bagging does — resample the rows, fit in parallel, average — and you get 8 MPa off with a steadier hand. Resampling can't change what a procedure systematically misses, so bias walks through the average untouched. That's the wall, and the reason a second family exists.

The way out is almost rude in its directness. Don't average the four-leaf trees. Line them up, and give each one the job of predicting what the ones before it got wrong.

Think of It Like This

How It Actually Works

One sentence separates the two families

Bagging fits models in parallel on resampled data and averages them, which cuts variance and leaves bias where it was. Boosting fits models in sequence, each one targeting what the previous ones got wrong, which cuts bias and can push variance up.

That's the taxonomy; the rest is detail hanging off it. Keep the bias–variance tradeoff in view and the choice is obvious. A deep tree on 1,000 batches is nearly all variance, so bag it. A four-leaf tree is nearly all bias, so boost it.

Why the learners have to be weak

Boosting runs on weak learners, usually depth-2 or depth-3 trees. Odd, until you notice the mechanism removes one thing only: a stump is high bias and low variance, badly wrong but reliably so, and reliable wrongness is what a correction step can chase.

Hand it a fully grown tree and there's nothing to do — round one fits the training set almost exactly, so round two is handed near-zero errors, which is noise, and fits that. Two to six levels, no deeper.

The additive model, and its two dials

After $M$ rounds the model is a sum, not a vote:

$F_0$ is the starting guess, usually the mean strength across all batches. Each $h_m$ is a weak learner fitted to what $F_{m-1}$ still got wrong. $\eta$ is the learning rate, a number like 0.1 that shrinks every contribution before it's added, and $M$ is the round count.

Those two trade almost directly: halving $\eta$ roughly doubles the $M$ you need for the same fit, since each round moves you half as far. A small rate with many rounds beats the reverse, so fix $\eta$ low and let validation error decide $M$.

Which is the real cost of going sequential. A forest converges: add trees and the curve flattens. Boosting has no floor, and keeps going until it's fitting the noise in your crushed cylinders, so early stopping on unseen data is part of the method, not a precaution.

One more family, briefly. Stacking fits several different models — a forest, a boosted ensemble, a linear fit — then trains a small meta-model on their predictions to learn how to blend them. It wins competitions and stays rare in production, because those base predictions have to be out-of-fold. Feed it predictions the base models made on their own training rows and it learns to trust whichever memorised hardest.

Show Me the Code

One-dimensional concrete: strength against a single mix ratio, boosted with stumps. Watch the rate and the round count trade.

1import numpy as np2
3rng = np.random.default_rng(1)4mix = rng.uniform(0.0, 1.0, 300)5strength = 40.0 + 12.0 * np.sin(6.0 * mix) + rng.normal(0.0, 2.0, 300)  # 2 MPa of test noise6
7def boost(rate: float, rounds: int) -> float:8    pred = np.full(300, strength.mean())          # round 0 predicts the yard average9    for _ in range(rounds):10        r = strength - pred                       # what the ensemble still gets wrong11        cuts = np.quantile(mix, np.linspace(0.05, 0.95, 19))12        fits = [np.where(mix <= c, r[mix <= c].mean(), r[mix > c].mean()) for c in cuts]13        pred = pred + rate * fits[int(np.argmin([((r - f) ** 2).sum() for f in fits]))]14    return float(np.sqrt(((strength - pred) ** 2).mean()))15
16print(" | ".join(f"rate {a} x {b} rounds -> {boost(a, b):.2f}" for a, b in ((0.1, 100), (0.05, 200), (0.5, 400))))17# -> rate 0.1 x 100 rounds -> 2.21 | rate 0.05 x 200 rounds -> 2.22 | rate 0.5 x 400 rounds -> 1.98

Halving the rate and doubling the rounds lands within 0.01 MPa of the same fit. The third run is the warning: training error 1.98 is below the 2 MPa of noise the cylinders carry, so those rounds bought memorisation.

Watch Out For

The Quick Version

• Bagging: parallel, on resampled rows, averaged. Cuts variance. Boosting: sequential, each learner fitted to the leftover error. Cuts bias.
• Boosting wants weak learners, depth 2 to 6. Boost a deep tree and it overfits in a few rounds: after round one there's nothing left but noise.
• The model is a sum of shrunken contributions rather than a vote, so order is load-bearing and rounds can't be parallelised.
• Learning rate and round count trade directly: halve the rate, roughly double the rounds. Small rate, many rounds, wins.
• Unlike a forest, boosting will overfit given enough rounds. Early stopping on held-out data is part of the method.
• Stacking is the third family: a meta-model blending diverse models. It leaks unless the base predictions are out-of-fold.

What to Read Next

• AdaBoost is the original, and the one where you can watch the weights move.
• Gradient Boosting generalises all of this to any differentiable loss.
• Bagging is the other half of the contrast.
• Random Forests is what to reach for when you want no learning rate.
• Bias–Variance Tradeoff is the frame that makes the family split obvious.
• Definitions worth a look: Variance and Data Leakage, the one stacking walks into.