Skip to content
AI360XpertBeta

Deep Ensembles

Train the same network five times from different starting weights and let the members vote. When they agree you can answer, when they split you are guessing.

Five member votes of 0.99, 0.15, 0.85, 0.30 and 0.60 average to 0.58 with variance 0.10, flagging doubt where agreement at 0.93 does not
Five member votes of 0.99, 0.15, 0.85, 0.30 and 0.60 average to 0.58 with variance 0.10, flagging doubt where agreement at 0.93 does not

Why Does This Exist?

A single network never says it is lost. Train a cat against dog classifier, show it an airplane, and you get back dog at 99 percent. The weights just extend their old boundary into empty space. That false certainty is what breaks deployments, not plain errors.

Two ideas make the rest click. A classifier turns raw scores into probabilities with a softmax, and the top probability is the stated confidence. Variance measures how far a set of numbers spreads around its mean. If you have not priced right against wrong yet, start with classification metrics.

This page covers the voting fix: train several nets, average them, and read their spread as doubt. The split behind that spread lives in aleatoric vs epistemic uncertainty. The gate that blocks strange inputs before they reach your model lives in out-of-distribution detection. The repair for shaky numbers on familiar inputs lives in calibration.

Think of It Like This

Five friends finishing your sentences

You hum half a chorus everyone knows. Five friends shout the next line together, word for word. You trust it.

Now you mumble half a sentence in a language none of them speaks. One guesses a lyric, one guesses an order at a cafe, one guesses a football chant, one shrugs toward the weather, one repeats your mumble back louder. Nobody agrees. You do not need to grade each guess. The split alone tells you nobody knows.

Each friend is one trained member, their different lives are the different starting weights, the known chorus is training distribution, the foreign mumble is the new photo, and the spread of completions is the variance.

Where it stops: friends can say they do not know, and they share one language. Network members always output numbers and never talk. You have to add the rule that turns a wide spread into hold back.

How It Actually Works

Call the input xx, meaning the photo or row you score. Call the label yy, meaning dog or not. Call θm\theta_m the weights of member mm, meaning one fully trained network. Write pmp_m for P(y∣x;θm)P(y \mid x; \theta_m), meaning the dog probability member mm reports. With MM members, usually five, the ensemble prediction is the plain average:

pˉ=1M∑m=1Mpm\bar{p} = \frac{1}{M} \sum_{m=1}^{M} p_m

Doubt is the population variance of those votes, meaning the mean squared gap from the average:

Var=1M∑m=1M(pm−pˉ)2\mathrm{Var} = \frac{1}{M} \sum_{m=1}^{M} \left(p_m - \bar{p}\right)^{2}

Training is deliberately boring. Same architecture, same data, same loss. The only differences are random starting weights and a different shuffling of minibatches. That is enough to send each optimizer down a different path. Unlike bagging, there is no bootstrap resample. Unlike dropout at test time, the members are full networks, not masks of one net. The full split of error into bias, variance, and noise lives in the bias-variance tradeoff.

Worked example: one strange photo, five votes

Use a dog probability, so 1.00 means sure dog and 0.00 means sure not dog. A strange photo draws:

p=[0.99,0.15,0.85,0.30,0.60]p = [0.99, 0.15, 0.85, 0.30, 0.60]

Three say dog, two say not dog. Add them:

0.99+0.15+0.85+0.30+0.60=2.890.99 + 0.15 + 0.85 + 0.30 + 0.60 = 2.89

Divide by five:

pˉ=2.89/5=0.578≈0.58\bar{p} = 2.89 / 5 = 0.578 \approx 0.58

The mean still leans dog, which is why the mean alone would fool you. Now the gaps from 0.578 are 0.4120.412, −0.428-0.428, 0.2720.272, −0.278-0.278, and 0.0220.022. Square each:

0.169744+0.183184+0.073984+0.077284+0.000484=0.504680.169744 + 0.183184 + 0.073984 + 0.077284 + 0.000484 = 0.50468

Divide by five:

Var=0.50468/5≈0.1009≈0.10\mathrm{Var} = 0.50468 / 5 \approx 0.1009 \approx 0.10

What agreement looks like for contrast

The same five nets on a familiar dog photo vote [0.92,0.95,0.90,0.93,0.94][0.92, 0.95, 0.90, 0.93, 0.94]. The sum is 4.644.64, so the mean is 4.64/5=0.928≈0.934.64 / 5 = 0.928 \approx 0.93. The squared gaps sum to 0.001480.00148, so variance is 0.00148/5≈0.00030.00148 / 5 \approx 0.0003. Same rule, same arithmetic. One case reads 0.10, the other 0.0003, about a 340 fold gap. That gap is the diagram headline: five boxes on the left, one mean and one spread on the right, and a chip that says doubt only when the spread is wide.

Why the diversity has to be real

The variance trick rests on one assumption: members must be genuinely different. Different starts plus shuffled batches give you that. Same seed, same order, same pretrained checkpoint with no noise, and the five paths collapse into one. Then disagreement reads near zero while ignorance stays high. Check overfitting and underfitting first too, because this whole page assumes trained members, not broken ones.

Code

The point is the check: rebuild both cases from the worked example and confirm 0.58 with 0.10 against 0.93 with 0.0003.

import numpy as np
strange: np.ndarray = np.array([0.99, 0.15, 0.85, 0.30, 0.60])familiar: np.ndarray = np.array([0.92, 0.95, 0.90, 0.93, 0.94])
mean_strange: float = float(strange.mean())var_strange: float = float(strange.var())mean_familiar: float = float(familiar.mean())var_familiar: float = float(familiar.var())
print(f"{mean_strange:.2f} {var_strange:.2f}")  # -> 0.58 0.10print(f"{mean_familiar:.2f} {var_familiar:.4f}")  # -> 0.93 0.0003

The printout matches the hand computation: strange photo mean 0.58 with spread 0.10, familiar photo mean 0.93 with spread 0.0003. Change one strange vote from 0.15 to 0.90 and watch the variance fall by more than half. That sensitivity is the feature.

Watch Out For

All five agree and are still wrong

Five members share the same data, the same architecture, and the same blind spot. A snow covered stop sign, a new slang phrase, or a sensor drift walks in, and every member votes dog at 0.95. Disagreement sits near zero, so the gate stays open and the wrong answer ships with full confidence.

Symptom: near zero variance on inputs that later prove foreign, often in clusters by time or source. Fix: never read agreement as correctness on foreign inputs. Put the gate from out-of-distribution detection in front, keep starts and shuffles different so agreement is earned, and abstain only for inputs that pass the gate.

Reading the mean as a calibrated chance

The strange photo averages to 0.58 and a dashboard shows 58 percent. A team treats that as a calibrated rate and auto approves near that cut. But 0.58 from a 3 to 2 split is not the same as 0.58 from five votes near 0.58. One is conflict, the other is consensus.

Symptom: the 0.50 to 0.65 bucket misses far more often than its label says, while accuracy looks fine elsewhere. Fix: route wide spread inputs to review or to the uncertainty split in aleatoric vs epistemic uncertainty, and fix familiar input numbers with calibration instead of trusting the raw average.

The Quick Version

  • Deep ensembles train the same network five times from different random starts with shuffled batches, then average the five probabilities.
  • Our strange photo draws 0.99, 0.15, 0.85, 0.30 and 0.60, so the mean is 2.89 divided by 5, about 0.58.
  • The population variance of those five votes is 0.50468 divided by 5, about 0.10, which flags doubt.
  • The same five nets on a familiar dog photo give mean about 0.93 with variance about 0.0003, so the strange case spreads about 340 times wider.
  • Shared blind spots still fool all five at once, so agreement alone never proves an input belongs to the training world.
  • Five members cost about five times the training and serving, so teams pay it for abstention and data choice, not for every low stakes call.