Dropout

Zero out a random slice of a layer on every training pass, and one set of weights quietly trains a huge family of thinned networks that vote at inference.

Four units train in two different thinned lineups while the expected sum holds at 3.2, because dividing each survivor by 1 minus p leaves inference needing no rescaling

Why Does This Exist?

Twenty-eight thousand product reviews, one hidden layer of 512 units, training accuracy 99.4%. Reviews from last month arrive and it manages 84%. Nothing crashed. The layer learned those particular reviews instead of the language they were written in, which is overfitting by the numbers: a 15-point gap between memorized and new.

A hidden layer is a bank of detectors: each unit fires when it spots something worth reporting, and the next layer reads all 512 reports to make the call. With that many free detectors against 28,000 examples, memorization fits comfortably. But "it memorized" is too vague to fix. The specific failure is that units gang up: unit 41 fires on the word "not", useless alone as sentiment, while unit 88 learns a correction that means something only when unit 41 fired. Together they nail 300 training reviews. That is a co-adapted pair, a private agreement that holds on the training set and snaps everywhere else.

This page covers the fix that breaks those agreements. The normalizer that steadies layers a different way is batch normalization. The penalty that shrinks weights instead of removing units is weight decay.

Think of It Like This

The kitchen where a third of the line calls in sick

Every service, a random third of your cooks fails to show, and you learn which third only when tickets start printing.

Run a kitchen that way for a year and nobody can be the only person who knows the sauce. Four people learn it instead, so recipes stop living in individuals and start living in the room. You are also not running one kitchen: each service is a different subset of the same staff, sharing one recipe book that every service improves.

Then the night the critic books, everybody shows up. Full line. Which is fine, except this kitchen never cooked at full strength and is about to push out half again as much food as any training night. Somebody had better have thought about portions.

The mapping: cooks are units, the missing third is the drop rate, the shared recipe book is the one weight matrix, and the portions problem is the rescaling inference needs. Where it stops: cooks notice absence and compensate, while surviving units never know who is missing. The robustness comes from the sampling, not from anyone adapting in the moment.

How It Actually Works

Sampling a thinned network on every pass

On each forward pass during training, every unit flips its own coin. With probability $p$ , the drop rate, that unit's output becomes zero for this pass, independently per unit and per pass, and the backward pass routes gradient only through the survivors. A layer of $n$ units has $2^n$ on-off patterns, so 512 units means $2^{512}$ thinned networks, and training visits a random handful, one per batch, each visit updating the same shared weights.

That is bagging with the expensive part removed. Bagging fits many independent models on resampled data and averages them, and it works because unshared errors cancel. Dropout's members share weights instead of training separately, and they are never averaged explicitly: one full forward pass approximates the ensemble.

Now the co-adaptation story has teeth. Unit 88 cannot build a correction leaning on unit 41, because unit 41 is missing on a large share of the passes where unit 88 is present. Any strategy needing a specific partner gets punished, so what survives is a representation spread across the layer.

Why inference needs no adjustment

Say each of 4 units outputs 0.8 and the drop rate is $p = 0.2$ . The keep probability is $1 - p = 0.8$ , and each survivor is divided by it, reporting $0.8 / 0.8 = 1.0$ . The expected sum over the coin flips:

4 \times 0.8 \times 1.0 = 3.2 = 4 \times 0.8

Exactly the sum with no dropout at all. In general:

\mathbb{E}\big[\tilde{h}_i\big] = \frac{(1-p)\,h_i}{1-p} = h_i

$h_i$ is what unit $i$ would normally output and $\tilde{h}_i$ is what it reports after dropout. That division during training is inverted dropout, and it is why model.eval() is load-bearing rather than housekeeping: it switches the masks off, and no rescaling is owed because training already paid it.

Rates, and where it still earns a place

Inside transformer blocks, 0.1 to 0.3 on attention weights and feed-forward outputs. Historically 0.5 in wide fully connected layers, where dropout started and where that number comes from. Scale the rate with the width: dropping half of a 32-unit layer removes real capacity.

Be honest about the trajectory. In convolutional vision dropout has largely been displaced: batch normalization and augmentation do the regularizing, and zeroing individual activations inside a spatially correlated feature map was never a clean fit. In transformer blocks it is still standard, and it composes with weight decay, which pulls on how large weights are rather than which units are present.

Code

The point is the check: the expected sum holds at 409.6 for every rate while the per-pass spread grows.

1import random2import statistics3
4
5def dropout(h: list, p: float, training: bool, rng: random.Random) -> list:6    if not training:7        return list(h)  # inference: masks off, nothing to undo8    scale = 1.0 / (1.0 - p)9    return [0.0 if rng.random() < p else v * scale for v in h]10
11
12rng = random.Random(0)13h = [0.8] * 512  # one hidden layer, every unit mildly active14
15for p in (0.0, 0.3, 0.5):16    passes = [sum(dropout(h, p, True, rng)) for _ in range(2000)]17    ev = sum(dropout(h, p, False, rng))18    print(f"p={p}: train sum {statistics.mean(passes):6.1f} +/- {statistics.pstdev(passes):4.1f}, eval sum {ev:6.1f}")19    # -> p=0.0: train sum  409.6 +/-  0.0, eval sum  409.620    # -> p=0.3: train sum  409.8 +/- 11.5, eval sum  409.621    # -> p=0.5: train sum  409.8 +/- 18.3, eval sum  409.6

The mean holds near 410 at every rate while the spread climbs from 0.0 to 18.3. That noise is the regularization, and it is why dropout usually needs more epochs before the loss settles.

Watch Out For

Evaluating a model that is still in training mode

The tell is a validation number that moves between two runs on identical data: 0.883, rerun unchanged, 0.891. That is not data noise, it is dropout still firing, a random share of every layer missing from every prediction, so the score reads worse than the model deserves.

Call model.eval() before the validation loop and model.train() after. Wire the switch into the evaluation function rather than trusting memory, because there is no exception and no warning. The same switch freezes batch-norm statistics, so forgetting it costs twice.

Reading a training loss above the validation loss as a leak

Training loss 0.34, validation loss 0.29, and the hunt for data leakage begins. Almost always there is nothing to find.

Training loss is measured on the thinned network with part of every layer switched off; validation loss is measured on the whole network. Different models, so the inequality says nothing about leakage. To compare fairly, score the training set again in eval mode first. If it still looks strange, then go looking.

The Quick Version

Each unit is zeroed independently with probability $p$ on every training pass, so every batch trains a different thinned network.
A layer of $n$ units defines $2^n$ subnetworks sharing one weight matrix, and the full network at inference approximates their average.
A cheap relative of bagging. The mechanism is anti-co-adaptation: no unit can lean on a specific partner.
Inverted scaling divides survivors by $1-p$ during training, which is why inference needs no adjustment.
Four units at 0.8 with $p = 0.2$ hold an expected sum of 3.2 either way, and 512 such units hold 409.6 with spread 11.5 at $p = 0.3$ .
Eval mode is not optional. Forgetting it makes evaluation noisy and pessimistic, and it freezes batch-norm statistics too.