Core ML

AdaBoost

Fit a weak learner, see which rows it got wrong, make those rows count for more, refit. The vote weights and the row weights are both one line of algebra.

The examples the first stump misclassified carry more weight into the second round, so the next stump moves its cut to cover them

Why Does This Exist?

In 1990 this was a theory question: if all you can build is a rule that beats a coin flip, can you combine enough of them into a rule as accurate as you like? AdaBoost, from Freund and Schapire in 1995, was the answer, and it also happened to work on real data.

Here's the case we'll carry down the page. A processing line sorts walnuts and the hollow ones have to come off the belt. Two cheap sensors per nut: weight in grams, and the ringing frequency when a paddle taps it. Hollow nuts are lighter and ring higher, but not reliably — shell thickness varies and the belt shakes.

One cut on weight gets you 70% correct. So does one cut on frequency. Neither is worth installing, and nobody is buying you a better sensor.

Boosting says to run those cuts in sequence, each fitted to what the last ones got wrong. AdaBoost was the first version that worked: instead of handing the next learner an error to predict, it hands it the same data with different weights, heavier on the nuts the sequence keeps missing.

Think of It Like This

A revision deck where missed cards go back on top

You're working through a deck of flashcards. Get one right and it goes to the bottom. Get one wrong and it goes back on top, so you see it again within a few cards. The deck reorganises itself around your weak spots without you deciding anything.

That's the whole scheme. Nothing is discarded, nothing is duplicated, the order and frequency of what you see changes.

Now suppose one card is misprinted, its answer flatly wrong. You'll never get it right, so it returns to the top every pass, forever, and soon you're revising one broken card. Hold on to that image — it's the failure mode that pushed the field past this algorithm.

How It Actually Works

One round, by hand

Every nut starts with weight $1/n$ , so ten nuts start at 0.10 each. Then, each round:

Fit a stump on the weighted data, which means picking the cut that minimises weighted error rather than plain error. Score it: $\varepsilon$ is the total weight on the nuts it misclassified. Then give the stump its vote:

\alpha = \tfrac{1}{2}\ln\frac{1 - \varepsilon}{\varepsilon}

$\alpha$ is how loudly this stump speaks in the final vote. At $\varepsilon = 0.5$ the logarithm's argument is 1, so $\alpha = 0$ and a coin-flip stump is ignored; as $\varepsilon$ falls, $\alpha$ climbs without limit. Worse than a coin flip gives a negative $\alpha$ , which reads the stump backwards.

Then multiply the weight of every misclassified nut by $e^{\alpha}$ , every correct one by $e^{-\alpha}$ , and renormalise so the weights sum to one.

Ten nuts, three misclassified. $\varepsilon = 0.3$ , so $\alpha = 0.42$ . Those three go from 0.10 to 0.167 each and the seven good ones drop to 0.071. Add up the three: 0.50. Not a coincidence — the update always leaves the mistakes holding exactly half the weight, so the next stump faces a problem where the last one's failures count as much as everything else together.

The reframe that produced gradient boosting

For five years this was a reweighting trick with a proof attached. Then Friedman, Hastie and Tibshirani showed in 2000 what it had been doing all along: AdaBoost is coordinate-wise gradient descent on exponential loss, in function space rather than parameter space. Each round picks the weak learner that reduces

L(y, F) = e^{-y F(x)}

fastest, and takes a step in that direction. $y$ is the label written as $-1$ or $+1$ , $F(x)$ is the running weighted vote, and the product $yF(x)$ is the margin — positive when the ensemble is right, large when it's confidently right.

Nothing about the mechanism changed; the shape of it did. If the reweighting is a gradient step on one particular loss, the loss becomes a parameter. Swap the exponential for anything differentiable and you have gradient boosting, which is what the field did almost immediately.

What the exponential loss costs

Look at $e^{-yF}$ for a nut that's wrong. Margin $-1$ costs 2.7, margin $-3$ costs 20, margin $-5$ costs 148. The penalty on a confidently-wrong example grows without bound, and the weight update inherits that shape.

For a borderline nut that's the behaviour you want. For a nut a tired inspector labelled hollow when it wasn't, it's a disaster: nothing will ever get it right, so its weight compounds by $e^{\alpha}$ every round until later stumps fit almost nothing else. AdaBoost's sensitivity to label noise isn't an implementation quirk. It's the loss, working as designed.

Show Me the Code

The whole update, on the ten-nut round above.

1import numpy as np2
3def one_round(w: np.ndarray, wrong: np.ndarray) -> tuple[float, np.ndarray]:4    err = float(w[wrong].sum())                     # weighted error, not a plain miss count5    alpha = 0.5 * np.log((1.0 - err) / err)         # this stump's vote, rising as err falls6    w = w * np.exp(np.where(wrong, alpha, -alpha))  # mistakes up, correct ones down7    return alpha, w / w.sum()                       # renormalise, so the weights stay a distribution8
9missed = np.zeros(10, dtype=bool)10missed[[2, 5, 9]] = True                            # the first stump calls three of ten wrong11alpha, w = one_round(np.full(10, 0.1), missed)12print(f"vote {alpha:.3f} | missed nut {w[missed][0]:.3f} each | sound nut {w[~missed][0]:.3f} each | mistakes hold {w[missed].sum():.3f}")13# -> vote 0.424 | missed nut 0.167 each | sound nut 0.071 each | mistakes hold 0.500

Four lines of arithmetic, and that's AdaBoost. Refit the stump on those weights, collect the next $\alpha$ , repeat a few hundred times, classify by the sign of the $\alpha$ -weighted vote. Viola and Jones built the first real-time face detector out of exactly this in 2001, the weak learners being single light-and-dark rectangle patterns.

Watch Out For

Running AdaBoost on data with mislabelled rows

The tell is specific. Training error keeps falling, round after round, while validation error bottoms out somewhere around round thirty and then climbs steadily. Not a plateau. A climb.

That's the misprinted flashcard. A handful of wrongly labelled nuts each pick up a factor of $e^{\alpha}$ per round, and once a few rows hold most of the weight every later stump fits them and nothing else. Check whether your heaviest rows are mislabelled — AdaBoost sorts them to the top for you — and if labels are noisy at all, boost a loss that doesn't compound: log loss, Huber, squared error. Cleaning labels beats any hyperparameter here.

Reaching for AdaBoost because it's the boosting algorithm you know

It's the one in the textbook chapter, so it's the one that gets tried. The result is usually fine, and beaten by a default-configured gradient-boosted tree on the same data.

A modern implementation gives you a choice of loss, second-order gradients, regularisation, missing-value handling and early stopping. AdaBoost gives you exponential loss and stumps. Read this page for the mechanism it invented and the reframe it triggered, then reach for a gradient-boosted tree library when something ships.

The Quick Version

Fit a weak learner on weighted data, score its weighted error $\varepsilon$ , give it vote $\alpha = \tfrac{1}{2}\ln((1-\varepsilon)/\varepsilon)$ , multiply the mistakes' weights by $e^{\alpha}$ , renormalise, repeat.
The update always leaves the misclassified rows holding exactly half the total weight.
$\alpha$ is zero for a coin-flip learner and negative for a worse one, which flips its vote. Predictions are the sign of the $\alpha$ -weighted total.
Invented as a reweighting scheme, later shown to be coordinate-wise gradient descent on exponential loss in function space.
That reframe produced gradient boosting: keep the mechanism, swap the loss.
Exponential loss grows without bound in the negative margin, so one mislabelled row can capture the ensemble — the reason AdaBoost is no longer the default.