Skip to content
AI360Xpert
Visual Explainers
Visual explainer

Gradient Boosting

Chain weak learners so each one corrects only the mistakes of the one before it — an additive process that converts many shallow trees into a powerful ensemble.

One weak learner makes a prediction and leaves large residual errors pointing in many directions.
One weak learner makes a prediction and leaves large residual errors pointing in many directions.

A single shallow tree can only approximate the data coarsely. What is left over — the gap between prediction and truth — is the residual. Gradient boosting treats that residual as the next problem to solve.

Train on the Mistakes

A second learner is trained only on the residuals left by the first, correcting what was missed.
A second learner is trained only on the residuals left by the first, correcting what was missed.

The second tree never sees the original labels. It fits the residuals from the first tree, shifting the ensemble prediction closer to the truth. Then a third tree fits the residuals of the combined prediction, and so on.

An Additive Chain of Corrections

Each stage adds a small correction to the sum, building up the final prediction as a chain of fixes.
Each stage adds a small correction to the sum, building up the final prediction as a chain of fixes.

The final prediction is the plain sum of every tree's output. Each tree is deliberately kept shallow — a weak learner — so no single stage can overfit. The strength comes from the chain, not from any individual component.

Learning Rate Keeps Steps Small

Shrinking each correction by a learning rate prevents over-correcting and keeps the ensemble stable.
Shrinking each correction by a learning rate prevents over-correcting and keeps the ensemble stable.

Each stage's correction is multiplied by a learning rate before it is added. A rate near 1 risks overshooting the target; a rate near 0.1 takes many small steps, which regularises the ensemble and usually lands closer to the true minimum.

Where It Breaks

Adding too many stages drives training error to zero but test error climbs as the model memorises noise.
Adding too many stages drives training error to zero but test error climbs as the model memorises noise.

There is no natural stopping point in the residual-fitting loop. Training error always falls; test error eventually turns around. Adding stages past the optimal count is gradient boosting's failure mode — it will memorise every quirk of the training set if allowed to run long enough. Early stopping and cross-validation are not optional.

The Quick Version

  • One shallow tree makes a coarse prediction; the gap is the residual.
  • The next tree is trained to predict the residual, not the original label.
  • Every stage adds its scaled correction to a running sum.
  • A learning rate below 1 prevents over-correction at each step.
  • Too many stages always overfits — use early stopping, not faith.

Related Concepts