Core ML

Bagging

Fit the same twitchy model to many resamples of your own data, then average. Variance falls fast, then stalls, because the resamples overlap so heavily.

Three draws with replacement from one dataset produce three different models whose average is steadier than any of them, and the rows each draw missed become its free validation set

Why Does This Exist?

Average enough noisy guesses and the noise cancels. That's the whole idea, and close to the only free thing in machine learning.

You're mapping the bed of a reservoir from 300 sonar soundings taken off the side of a boat, and you want a depth prediction anywhere on the surface. A deep decision tree handles it: carve the surface into blocks, predict the average sounding in each. It will also redraw those blocks completely if you take 300 different soundings. Same reservoir, same procedure, a visibly different map.

That instability has a name. Variance is how much your fitted model moves when the sample moves, with nothing about the truth having changed — see the bias–variance tradeoff. Bagging is the cheapest treatment anyone has found for it, and it asks for no new data.

Think of It Like This

Twenty strangers guessing the weight of a pumpkin

A stall at a county fair. Guess the pumpkin's weight, closest guess takes it home. Twenty strangers write down twenty numbers, some 18 kg high, some 14 kg low, and the average lands within a kilogram or two of the truth. Nobody in that queue was any good. The averaging was.

Now the stall owner props up a sign: last year's pumpkin weighed 64 kg. Everyone reads it on the way in. Every guess now clusters near 64, carrying the same pull in the same direction. Add a thousand people and you land in the same wrong place.

Independent errors cancel. Shared errors don't. Hold on to that sign, because it's the whole reason random forests had to be invented.

How It Actually Works

Draw, fit, average

Bootstrap aggregating, contracted to bagging. Draw $B$ samples from your 300 soundings, each also of size 300, with replacement. Fit an independent model on each. Average the $B$ predictions for a numeric target, or take the majority vote for a class. There is no step four.

With replacement is the part that gets skimmed, and it's the only reason any of this works. Draw 300 rows without replacement from 300 rows and you get the same 300 rows and $B$ identical models. With replacement, a row can appear twice, three times, or never, so each model sees a slightly different reservoir. Bootstrap and resampling covers why sampling your own sample is legitimate.

Why variance falls, and where it stops

Average $B$ independent estimators that each have variance $\sigma^2$ — the spread of one model's prediction across resamples — and you get $\sigma^2/B$ . Thirty models, a thirtieth of the variance. If that held, you could buy your way to zero.

It doesn't hold. Any two bootstrap samples come out of the same 300 soundings and share most of their rows, so the models they produce make correlated mistakes. Write $\rho$ for the correlation between two models' predictions. The variance of the average is then

\rho\,\sigma^2 + \frac{1 - \rho}{B}\,\sigma^2

Push $B$ to infinity and the second term disappears. The first stays exactly where it is: $\rho\sigma^2$ is a floor set by how alike your models are, and no quantity of them gets you underneath it. At $\rho = 0.6$ you can remove at most 40% of the variance, whatever you spend on compute. That floor is why random forests exists — it goes after $\rho$ , not $B$ .

Notice what never appears in that expression: bias. Every model in the bag comes from the same procedure, so whatever it gets systematically wrong, all of them get wrong in the same direction and the average preserves it perfectly. Bagging is a variance instrument and nothing else — enormous for deep trees, nearly pure variance themselves; worth roughly nothing for a linear model, nearly pure bias.

The rows you missed

Each resample draws $n$ times from $n$ rows, so the chance a given row is never picked is $(1 - 1/n)^n$ , which converges to $e^{-1} = 0.368$ . So a resample holds about 63.2% of the rows and leaves 36.8% behind.

The ones left behind are out-of-bag for that model, and they're a gift: it never saw them, so its predictions on them are genuinely held out. Across the whole ensemble, every row has a set of models for which it was out of bag; average those predictions per row and score them. That's a validation estimate from the same data, with nothing set aside and no extra fitting.

Show Me the Code

A one-nearest-sounding predictor is about as unstable as a model gets — a single sounding decides the answer. Bag thirty of them.

1import numpy as np2
3rng = np.random.default_rng(3)4
5def survey(n: int = 60) -> tuple[np.ndarray, np.ndarray]:6    x = rng.uniform(0.0, 30.0, n)7    return x, 12.0 * np.sin(x / 6.0) + rng.normal(0.0, 2.0, n)  # reservoir bed, plus sonar noise8
9def nearest(x: np.ndarray, y: np.ndarray, at: float) -> float:10    return float(y[np.abs(x - at).argmin()])  # twitchy on purpose: one sounding decides11
12def bagged(x: np.ndarray, y: np.ndarray, at: float, b: int = 30) -> float:13    rows = rng.integers(0, x.size, (b, x.size))  # every resample is size n, with replacement14    return float(np.mean([nearest(x[r], y[r], at) for r in rows]))15
16one = [nearest(*survey(), 14.0) for _ in range(300)]17many = [bagged(*survey(), 14.0) for _ in range(300)]18kept = np.unique(rng.integers(0, 60, 60)).size / 6019print(f"single {np.var(one):5.2f}   bagged {np.var(many):5.2f}   rows used {kept:.2f}")20# -> single  4.48   bagged  2.04   rows used 0.60

Better than halved, and nowhere near thirty times better. The gap between $\sigma^2/B$ and what you got is the correlation floor, measured. The 0.60 is the $1 - e^{-1}$ figure turning up at $n = 60$ .

Watch Out For

Bagging a model that had no variance to give

You bag a linear regression, run it, and the ensemble scores within noise of the single model. The conclusion people reach is that ensembling was oversold.

The actual conclusion is that the base learner was wrong. A linear fit on 300 soundings barely moves when you resample it — its coefficients are essentially averages, and averages are stable. Nothing there for more averaging to cancel. Bagging pays in proportion to how unstable the thing you're bagging is, which is why the canonical base learner is a deep unpruned tree.

Resampling downstream of a fitted step, or across grouped rows

Two versions of one mistake. Fit your scaler, imputer or target encoder on all 300 soundings, then bootstrap the transformed matrix, and every model inherits the same fitted statistics. The resamples are less independent than they look, $\rho$ climbs, and out-of-bag error comes back optimistic. Every fitted step belongs inside the resample loop.

The grouped version is worse because it leaves no trace. If those 300 soundings came from 40 boat passes, several readings metres apart within each pass, then a row left out of a bag almost certainly has a near-twin inside it. The models correlate, the variance reduction silently fails to arrive, and out-of-bag error reports a number you could never reproduce on a fresh pass. Resample the passes, not the rows.

The Quick Version

Draw $B$ resamples of size $n$ with replacement, fit a model on each, average the predictions or take the vote.
Averaging $B$ independent models divides variance by $B$ . Bootstrap samples aren't independent, so you get $\rho\sigma^2 + (1-\rho)\sigma^2/B$ — more models shrink only the second term, and the first is a floor.
Bias passes through averaging untouched. Bag deep trees and high-variance learners; skip it entirely for linear models.
$(1-1/n)^n \to e^{-1}$ , so a resample contains roughly 63.2% of the rows and leaves 36.8% out of bag.
Those out-of-bag rows form a held-out set per model — a validation estimate without reserving any data.
Anything that correlates the base models — shared preprocessing, grouped rows, near-duplicates — removes the benefit without warning you.