Batch Normalization

Each channel is recentered with its own batch mean and spread, then handed a learned scale and shift. At test time a running average stands in because the batch is gone.

One channel reading 2, 4, 6, 8 is recentered to mean 0 and spread 1, then rescaled with scale 2 and shift 1.

Why Does This Exist?

Stack ten layers and each one trains on shifting ground. The first layer moves a little, so the second layer's input range slides. The second layer adapts, which slides the third layer's input, and so on down the stack. Steps that looked safe start overshooting, so you shrink the learning rate and training crawls.

This page covers the per-channel fix: recenter each channel with statistics taken across the batch, then let the layer relearn its preferred scale and shift. The batch-size side of the story lives in batch size effects, which explains why tiny batches make every estimate noisier. The per-example alternatives live in layer normalization and group normalization, which drop the batch dimension when it cannot be trusted.

You already know gradient descent nudges weights downhill from the gradient at hand. You also know activation functions bend each layer output before it travels onward. Batch normalization sits between those two ideas and steadies what the next layer receives before the bend. If either page feels fuzzy, skim it first, then come back.

Think of It Like This

The morning bread shelf

A bakery proofs four trays of loaves and they come out uneven: 2, 4, 6 and 8 centimeters tall. The baker lines them against the day average, trims every loaf to sit evenly around zero, then stretches the evened row to the shape customers like, tall with a small lift. One notebook average carries over for days when a single loaf arrives alone, and a tray of two loaves never gives a trustworthy average.

The tray is the batch, the day average and spread are the batch mean and spread, trimming even is standardizing, the chosen stretch is the learned scale and shift, the notebook is the running average, and the two-loaf tray is the small-batch failure.

Where it stops: loaves do not learn. The baker picks the stretch by taste, while the network picks its scale and shift by gradient. Nothing in the bakery explains why steadier inputs take larger steps.

How It Actually Works

Take one channel and the $m$ examples in front of you. Collect that channel's $m$ pre-activation values, meaning its raw sums before the bend. Subtract their mean, divide by their spread, then apply a learned scale and shift that belong to that channel alone.

The formula, per channel

Write $z_i$ for the channel value on example $i$ . Write $\mu_B$ for the batch mean and $\sigma_B^2$ for the batch variance, each taken across the batch $B$ for this one channel. Write $\epsilon$ for a tiny constant that keeps the division safe, $\gamma$ for the learned scale, and $\beta$ for the learned shift:

\hat{z}_i = \frac{z_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \qquad y_i = \gamma\,\hat{z}_i + \beta

The mean and variance pool across examples, never across channels. For a convolution the pool also covers every spatial position of that channel. The other cut, one example across its own features, is layer normalization, which never touches the batch dimension. That per-example habit is why sequence models prefer it.

Worked example with 2, 4, 6, 8

One channel reads $[2, 4, 6, 8]$ over $m = 4$ examples. The mean is:

\mu_B = \frac{2 + 4 + 6 + 8}{4} = 5.0

Deviations from the mean are $[-3, -1, 1, 3]$ , squares are $[9, 1, 1, 9]$ , and the variance is:

\sigma_B^2 = \frac{9 + 1 + 1 + 9}{4} = 5.0

The spread is $\sqrt{5.0 + 0.00001} \approx 2.24$ . Subtracting and dividing gives values centered at 0 with spread 1:

\hat{z} \approx [-1.34, -0.45, 0.45, 1.34]

Now rescale with $\gamma = 2$ and $\beta = 1$ , both picked by training, not by the formula:

y = 2\,\hat{z} + 1 \approx [-1.68, 0.11, 1.89, 3.68]

The diagram draws these same numbers: four raw readings, four centered readings, four rescaled outputs. Standardizing alone would pin every channel to the same shape, and the learned pair hands the shape back. Setting $\gamma = 1$ and $\beta = 0$ keeps the centered form, and training can learn exactly that when centered is best.

Training uses this batch, testing uses the running past

During training $\mu_B$ and $\sigma_B^2$ come from the batch in front of you, so one sample output depends on its batchmates. At test time there may be a single sample and no batch to average, so the layer uses running averages of the mean and spread collected while training. Same weights, different arithmetic, which is why train and test can disagree when the running past goes stale.

Steadier inputs take larger steps without overshooting, and that tolerance is where most of the speedup comes from. The price is batch dependence: with only a few examples per batch the mean and spread turn noisy, and the layer injects that noise into every forward pass. Batch size effects names the tradeoff, and group normalization is the per-example fallback when the batch stays small.

Code

The point is the check: recompute the hand example and confirm the centered values average to zero.

1import math2
3z: list[float] = [2.0, 4.0, 6.0, 8.0]4m: int = len(z)5mu: float = sum(z) / m6var: float = sum((v - mu) ** 2 for v in z) / m7std: float = math.sqrt(var + 1e-5)8hat: list[float] = [(v - mu) / std for v in z]9y: list[float] = [2.0 * v + 1.0 for v in hat]10
11print(f"{mu:.1f} {var:.1f} {std:.2f}")  # -> 5.0 5.0 2.2412print([round(v, 2) for v in hat])  # -> [-1.34, -0.45, 0.45, 1.34]13print([round(v, 2) for v in y])  # -> [-1.68, 0.11, 1.89, 3.68]14print(f"{sum(hat) / m:.1f}")  # -> 0.0

The printout matches the hand computation: mean 5.0, variance 5.0, spread 2.24, centered values averaging to 0.0, rescaled outputs from minus 1.68 to 3.68.

Watch Out For

Small batches turn the mean into noise

The same model trains calmly at a few hundred examples per batch, then wobbles or stalls at 8. The tell is that raising the batch size fixes it while tuning the learning rate does not. With 2 to 8 examples the per-batch mean and spread are guesses, and the layer feeds that guesswork into every output.

Symptom: jumpy loss, a gap between batch statistics and running averages, and dropout looking steadier by comparison. Fix: switch small-batch vision stacks to group normalization and sequence stacks to layer normalization, both of which compute statistics per example. One trap: gradient accumulation does not rescue batch norm, because the statistics still come from each small micro-batch.

Training looks fine while evaluation falls apart

Loss falls in training yet test predictions look broken. The cheap check is to push one batch through in training mode, then in evaluation mode, and compare. Wild divergence points at the running averages or a forgotten mode switch, not the weights.

Symptom: healthy training curves paired with bad held-out scores right from the first eval. Fix: confirm the model runs in evaluation mode at test time, give the running averages enough settled steps to converge, or freeze the weights and recompute the statistics with a few hundred forward passes. Backpropagation updates the weights, but only forward passes refresh the running past.

The Quick Version

Batch normalization recenters each channel with its own batch mean and spread, then applies a learned scale and shift per channel.
Values 2, 4, 6, 8 have mean 5.0 and spread about 2.24, center to minus 1.34, minus 0.45, 0.45, 1.34, and rescale with scale 2 and shift 1 to minus 1.68, 0.11, 1.89, 3.68.
Training uses the batch in front of you, so one output depends on its batchmates, while testing uses running averages kept during training.
Normalizing per example across features is layer normalization instead, which is why sequence models pick it.
Batches of 2 to 8 make the mean and spread noisy enough to harm training, so group or layer normalization replaces batch norm there.
Forgetting evaluation mode or shipping stale running averages breaks test scores while training curves look healthy.