Boosting

Shallow trees take turns fixing the leftover error, each added at half strength. The chain cuts bias that parallel averaging can't touch.

Two shallow stumps starting from a mean of 5.0 fix residuals in turn and cut squared error from 20 to 5

Why Does This Exist?

You price used cars from 1,200 past sales. Six numbers per car: age, mileage, engine size, doors, service stamps, prior owners. A depth 2 decision tree gives four prices and sits about 1,800 off on most rows, always low on large engines and always high on old city cars.

Averaging 500 such trees the bagging way steadies the quotes but keeps the same offset. That is the wall. Resampling can't change what a procedure misses in the same way each time. The split of error into bias and variance in the bias-variance tradeoff names it: you've hit bias, not variance.

So line the small trees up instead. Each one studies what the ones before it left wrong and adds a shrunken fix. This page covers that sequence idea only. AdaBoost covers the reweighting version for classes. Gradient boosting covers the gradient version that works with any loss.

Think of It Like This

Spot painting a fence

You paint a long fence with thin coats. The first coat goes on fast and leaves pale patches where the wood drank the paint. You don't repaint the whole fence. You walk back with a small brush and touch only the pale spots, with paint so thin it can't cover in one go.

Each pass alone looks bad. In order, each fixing what the last left pale, the fence turns solid. Order matters here. The third pass makes no sense without the first two under it.

Where it stops: past some point more passes hurt. Once the pale spots are gone you start painting the grain itself, and drips appear where no defect was. That's overfitting, and it's why you stop on held out boards, not when the fence looks good from your ladder.

How It Actually Works

The sum that never votes

After $M$ rounds the model is a sum, not a vote:

F_M(x) = F_0(x) + \eta \sum_{m=1}^{M} h_m(x)

Here $x$ means one row of features, $F_0$ means the starting guess, usually the mean of the target, each $h_m$ means one weak learner fitted to what the ensemble missed so far, $\eta$ means the learning rate, a small number like 0.5 or 0.1 that shrinks each fix before adding, and $M$ means the round count. Halve $\eta$ and you'll roughly double the $M$ needed for the same fit, since each step moves half as far. Small rate with many rounds wins, so you'll fix $\eta$ low and choose $M$ by held out error.

Why tiny trees do the work

Boosting wants stumps or depth 2 trees. A stump is high bias and low variance: badly wrong, but reliably so, and reliable wrongness is what a correction step can chase. Hand the sequence a deep tree and round one fits the training rows almost exactly, so round two gets near zero leftovers, which is noise, and fits that. Keep depth in the low single digits.

AdaBoost hands each stump the same rows with new weights, heavier on the rows missed so far. Gradient boosting hands each tree the residuals, meaning truth minus current prediction per row, or the gradient for other losses. Same loop, different definition of miss. Read those pages for each loss. Here you only need the loop.

Worked example: four rows from 5.0 to 3.5 and 6.5

Take four cars with size scores $x = [1, 2, 3, 4]$ and prices $y = [2, 4, 6, 8]$ in thousands. Start from the mean:

F_0 = 5.0 \text{ everywhere}

Residual $r$ means truth minus current prediction per row:

r = [-3, -1, 1, 3]

Squared error is $9 + 1 + 1 + 9 = 20$ . Fit a stump that splits small versus large, meaning $x \le 2$ against $x > 2$ . Left mean is $(-3 + -1) / 2 = -2$ . Right mean is $(1 + 3) / 2 = 2$ . So $h_1 = [-2, -2, 2, 2]$ . Add it at rate $\eta = 0.5$ :

F_1 = [4.0, 4.0, 6.0, 6.0]

New squared error is $4 + 0 + 0 + 4 = 8$ . One shrunken stump cut 20 to 8. Those are the top row numbers in the diagram.

Round two repeats on the new miss $[-2, 0, 0, 2]$ . Same split gives $h_2 = [-1, -1, 1, 1]$ . Add at 0.5:

F_2 = [3.5, 3.5, 6.5, 6.5]

Squared error is $2.25 + 0.25 + 0.25 + 2.25 = 5.0$ . So the chain runs 20 to 8 to 5. No leap, just two half strength fixes. Unlike a forest, there is no floor. Keep going and it fits noise, which is why early stopping on unseen rows is part of the method.

Code

The same two rounds in code. The point is the check: the means land where the hand arithmetic said, and the errors read 20.00, 8.00, 5.00.

1ys: list[float] = [2.0, 4.0, 6.0, 8.0]2
3
4def sq_error(truth: list[float], pred: list[float]) -> float:5    return sum((t - p) ** 2 for t, p in zip(truth, pred))6
7
8def stump_fix(resid: list[float]) -> list[float]:9    left: float = (resid[0] + resid[1]) / 2.010    right: float = (resid[2] + resid[3]) / 2.011    return [left, left, right, right]12
13
14rate: float = 0.515pred0: list[float] = [sum(ys) / len(ys)] * len(ys)16print(f"start {[f'{p:.1f}' for p in pred0]} error {sq_error(ys, pred0):.2f}")  # -> start ['5.0', '5.0', '5.0', '5.0'] error 20.0017
18resid1: list[float] = [y - p for y, p in zip(ys, pred0)]19fix1: list[float] = stump_fix(resid1)20pred1: list[float] = [p + rate * f for p, f in zip(pred0, fix1)]21print(f"round 1 fix {fix1} pred {[f'{p:.1f}' for p in pred1]} error {sq_error(ys, pred1):.2f}")  # -> round 1 fix [-2.0, -2.0, 2.0, 2.0] pred ['4.0', '4.0', '6.0', '6.0'] error 8.0022
23resid2: list[float] = [y - p for y, p in zip(ys, pred1)]24fix2: list[float] = stump_fix(resid2)25pred2: list[float] = [p + rate * f for p, f in zip(pred1, fix2)]26print(f"round 2 fix {fix2} pred {[f'{p:.1f}' for p in pred2]} error {sq_error(ys, pred2):.2f}")  # -> round 2 fix [-1.0, -1.0, 1.0, 1.0] pred ['3.5', '3.5', '6.5', '6.5'] error 5.00

The printout reads start 20.00, round one 8.00, round two 5.00, matching the hand path 20 to 8 to 5.

Watch Out For

Boosting trees that were already strong enough alone

You raise depth from 2 to 12 because deeper trees scored better solo. Training error hits near zero inside ten rounds and validation comes back worse than one pruned tree.

Symptom: the training curve collapses over a dozen rounds instead of easing down over hundreds, and round two onward fits leftovers near zero. Fix: cap depth at 1 to 3, drop the rate to 0.05, and choose rounds on held out rows with early stopping.

Picking rounds by training error

Training error under boosting falls by construction, since each round is fitted to lower that exact number. A curve still falling at round 2,000 says nothing about round 2,000 being good.

Symptom: you ship thousands of rounds because the training line kept sloping down, then new cars price worse. Fix: score each round on a validation split the sequence never sees, keep the best round rather than the last, and confirm small data choices with cross-validation schemes plus a train validation test split held aside.

The Quick Version

Boosting fits shallow trees in sequence, with each tree trained on the leftover error of the ensemble so far.
The ensemble is a sum starting from the mean, with each fix shrunk by the learning rate before adding.
Our four rows start at mean 5.0 with squared error 20, reach 4.0 and 6.0 with error 8 after one stump at rate 0.5, and reach 3.5 and 6.5 with error 5 after two.
Weak learners stay at depth 1 to 3, since deep trees leave only noise for later rounds to fit.
Learning rate and round count trade directly, so a small rate with many rounds chosen by held out error wins.
Boosting keeps fitting past the signal, so held out validation decides when to stop.