Visual explainer

AdaBoost

Trace classic binary AdaBoost through five illustrative samples, normalized reweighting and three signed stump votes, with training and noise limits.

A single threshold misses one of five illustrative training samples.

AdaBoost fits rules in sequence, raising the relative importance of mistakes before the next fit. Our synthetic samples have feature values 1 through 5 and labels +, +, −, +, −. A stump splits at one threshold. The first chosen cut, 2.5, misses sample 4; cut 4.5 ties, so we choose the smaller threshold.

Separate Samples From Votes

Round one gives each sample weight 0.20 and the stump vote 0.693.

Each sample starts at weight 0.20, giving weighted error 0.20. In classic binary AdaBoost, the stump's vote weight is half the natural log of (1 − error) / error: 0.693 here. Sample weights guide fitting; stump weights guide prediction. We use no shrinkage, with labels and predictions ±1.

Misses Get Louder

Reweighting raises the missed sample to 0.50 and normalizes the total to one.

Multiply the miss by exp(0.693) = 2 and each hit by exp(−0.693) = 0.5. Raw weights become 0.10, 0.10, 0.10, 0.40, 0.10. Divide by their total, 0.80: sample 4 gets 0.50 and each other sample gets 0.125. The normalized weights sum to one.

Refit on New Weights

The next stump moves its threshold to reduce the new weighted error.

Repeating cut 2.5 now costs 0.50. Refitting selects cut 4.5, which misses only sample 3 at weight 0.125 and earns vote 0.973. Both stumps make one unweighted mistake. Their different weighted errors explain the change; the fitting rounds depend on one another.

The Committee Wins

Three signed stump votes correctly classify all five training samples.

Round three reverses polarity at cut 3.5: minus on the left, plus on the right. Its weighted error is 3/14 and its vote is about 0.650. Add each stump's signed vote; the five scores are about 1.016, 1.016, −0.370, 0.929, −1.016. Their signs match the training labels.

Tune the Rounds

Zero training error does not determine the best number of rounds.

The committee makes 1, 1, then 0 training mistakes over these three rounds. We have no held-out outcomes here. More rounds cost more fitting and prediction work; choose rounds and base-model complexity using held-out evaluation. Three rounds solving this fixture supplies no universal stopping rule.

Where It Breaks

A wrong label can receive the same extra weight as a genuinely hard sample.

Suppose sample 4's observed plus label were wrong. The same first-round update would still raise its weight from 0.20 to 0.50. AdaBoost sees disagreement, not its cause. Noise can attract effort; high weight alone doesn't diagnose a bad label. This conditional example doesn't claim our clean fixture contains noise.

The Quick Version

One threshold misses one of the five illustrative training samples.
Sample weights guide fitting; stump vote weights guide prediction.
Reweight mistakes, then normalize all sample weights to one.
New sample weights change the next stump's best threshold.
Three signed stump votes fit this five-sample training set.
Zero training error doesn't establish the best round count.
Wrong labels can attract extra weight, too.