Gradient Boosting

Ordinary gradient descent nudges parameters. Gradient boosting nudges the prediction itself, one small tree per step, fitted to the gradient of your loss.

Starting at 5.0 against target 8, three shrunken fixes of 1.0, 0.8 and 0.5 climb to 6.0, 6.8 and 7.3, so squared error falls from 9.0 to 0.49

Why Does This Exist?

Boosting says fit each learner to what the last ones got wrong without defining wrong. For squared error, wrong means the residual, truth minus prediction. For skewed parcel counts, that definition puts a miss of 30 at 900 times the priority of a miss of 1, since $30^2 / 1^2 = 900$ . AdaBoost helps only for labels of minus one and plus one under exponential loss.

The fix from 2001 redefines wrong as the negative gradient of whatever loss you chose, one number per row. Squared error falls out as one case, log loss gives classification, Poisson handles counts without negative predictions, and quantile loss predicts a percentile for van sizing. One loop, many losses, which is why this page sits before gradient-boosted tree libraries.

Think of It Like This

Fixing a boomy hall with a measured sweep

A hall PA sounds muddy. Turning eight broad amplifier knobs by feel is working on the controls. Measuring instead, you play a sweep and read one number per frequency band saying this band is 6 dB too loud. You build a small correction tracing that shape and apply it at half strength, because the measurement is noisy and overshooting beats worse than undershooting. Then you sweep again.

Knobs are parameters, the per-band numbers are per-row gradients, the small correction is the tree, half strength is the learning rate, and the repeat sweep is the next round.

Where it stops: a hall has fixed acoustics and a meter you trust. Training data has sampling noise and shifting inputs, so the measured gradient is an estimate rather than a meter reading. Half strength protects a noisy estimate in a way a trusted meter would not need.

How It Actually Works

Descending on the prediction, not the knobs

Gradient descent steps parameters against the loss slope. Boosting steps the prediction $F(x)$ itself. For row $i$ at round $m$ :

r_{im} = -\left[\frac{\partial L(y_i, F(x_i))}{\partial F(x_i)}\right]_{F = F_{m-1}}

Here $L$ is your loss, $y_i$ the truth for row $i$ , $F_{m-1}$ the ensemble so far, and $r_{im}$ the direction that row wants to move. You cannot add 400 loose numbers to a model and generalize, so you fit a small tree $h_m$ to them and update $F_m = F_{m-1} + \eta h_m$ , with $\eta$ the learning rate.

For squared error $L = \tfrac{1}{2}(y - F)^2$ the derivative in $F$ is $-(y - F)$ , so the negative gradient is exactly the residual $y - F$ . Fitting residuals is therefore one loss substituted in, not the definition.

Worked example: 5.0 climbs to 7.3 toward 8

Target $y = 8$ . Start at the mean $F_0 = 5.0$ . Gap $8 - 5.0 = 3.0$ and squared error $3.0^2 = 9.0$ .

Round one fits the 3.0 gap, predicts $2.0$ of it, and adds half: $5.0 + 0.5 \times 2.0 = 6.0$ . New gap $8 - 6.0 = 2.0$ .

Round two reads $1.6$ of the 2.0 gap and adds half: $6.0 + 0.5 \times 1.6 = 6.8$ . New gap $8 - 6.8 = 1.2$ .

Round three reads $1.0$ of the 1.2 gap and adds half: $6.8 + 0.5 \times 1.0 = 7.3$ . Final gap $8 - 7.3 = 0.70$ and squared error $0.70^2 = 0.49$ .

Totals climb $5.0$ , $6.0$ , $6.8$ , $7.3$ while gaps shrink $3.0$ , $2.0$ , $1.2$ , $0.70$ . Error falls $9.0$ to $0.49$ . The diagram draws this same loop: running prediction, gradient, small tree, shrunken add-back.

Three dials

Learning rate $\eta$ scales every tree, so small means more rounds and smoother fits; 0.05 with early stopping beats tuning rounds at 0.3. Tree depth sets interaction order: depth 1 is additive, depth 3 lets three features combine, past 6 usually buys variance. Stochastic boosting fits each round on 50 to 80 percent of rows, decorrelating members the way a random forest does, which regularizes and runs faster.

Code

The point is the check: replay the 5.0 to 7.3 staircase above and confirm error falls 9.0 to 0.49.

1target: float = 8.02eta: float = 0.53
4f: float = 5.05for tree_reads in (2.0, 1.6, 1.0):6    f = f + eta * tree_reads7    print(f"{f:.1f} gap {target - f:.1f}")  # -> 6.0 gap 2.0, then 6.8 gap 1.2, then 7.3 gap 0.78
9print(f"{(8.0 - 5.0) ** 2:.1f} to {(8.0 - 7.3) ** 2:.2f}")  # -> 9.0 to 0.49

The printout matches the hand arithmetic: totals 6.0, 6.8, and 7.3 with gaps 2.0, 1.2, and 0.7, and squared error from 9.0 down to 0.49. Change $\eta$ to 1.0 and the first step lands at 7.0, overshooting faster, which is why half strength is the default seatbelt.

Watch Out For

Leaving the learning rate high and tuning rounds instead

Symptom: grids at 0.3 pick a small round count that wins validation but shifts meaningfully across seeds, with coarse steps that jump past the minimum and back.

Fix: set the rate to 0.05, turn on early stopping on validation, and let rounds run long. Smoother fit, less rerun variance, cost is compute only.

Boosting squared error onto skewed counts

Symptom: redelivery counts bounded at zero with rare 40s get predictions of 2.5 where truth is almost always 0 or 1, plus occasional negative counts, because $30^2 / 1^2 = 900$ tells the loop to chase tower blocks.

Fix: pick the loss from the target shape first. Poisson deviance fits counts on a log scale and stays positive, Tweedie covers long-tailed money with a spike at zero. One argument in every serious library, set before depth or rate.

The Quick Version

Gradient boosting is gradient descent in function space: the stepped object is the prediction, not a parameter vector.
Each round fits a small tree to per-row negative gradients and adds it scaled by the learning rate.
Totals 5.0, 6.0, 6.8, and 7.3 climb toward 8 while squared error falls from 9.0 to 0.49 in three half-strength steps.
For squared error the negative gradient is the residual, which is one case rather than the definition.
Any differentiable loss works: log loss for classes, Poisson for counts, quantile for intervals, Huber for outliers.
Small rate with early stopping, depth 3 to 6, and row subsampling of 50 to 80 percent is the working default.