Group Normalization

Split a layer's channels into fixed-size groups and normalise each group within one example, never across the batch, so a detector running two images per GPU normalises exactly as reliably as one running two hundred.

Batch statistics read down one channel across examples while group statistics read across two channels of one example, so values 2 and 4 become minus 1 and 1

Why Does This Exist?

A detector reading large scans fits two images per GPU before memory runs out. Batch normalization estimates its mean and variance down each channel across the batch, and with two samples that estimate swings run to run. Accuracy drops for no reason except the batch got small.

The batch-free fix of layer normalization goes too far the other way for convolutions: it pools every channel of the example into one shared mean, blending edge, texture, and color channels that detect different things. Group normalization splits the difference. Divide the channels into fixed groups and normalize each group inside one example only, so statistics never cross examples yet per-channel structure partly survives.

Think of It Like This

Grading one report card by subject groups

A teacher grades one student. Comparing her math score against thirty classmates works when the class is large and fails with three students: the curve depends on who else showed up. That is batch normalization.

Averaging her math, art, and gym scores into one number works with any class size but blends subjects that measure different things. That is layer normalization.

Grouping first fixes both: average sciences together and arts together, per student, with no other student involved. Each group mean is stable at class size three or three hundred, and math never dissolves into gym. That is group normalization with two groups.

Where it stops: report cards have natural departments while channels do not. Picking 32 groups is a tested default, not a law of the subject matter, and a bad group count still blends the wrong channels.

How It Actually Works

Split $C$ channels into $G$ groups of $C/G$ each. For one example and one group $g$ , gather its values across those channels and all spatial positions, then normalize:

\hat{x}_{i} = (x_i - \mu_g) / \sqrt{\sigma_g^2 + \epsilon}, \quad i \in g

Here $\mu_g$ and $\sigma_g^2$ are the mean and variance of group $g$ for this example only, and $\epsilon$ is a small constant for stability. No other example enters $\mu_g$ or $\sigma_g^2$ , so batch size 2 and batch size 200 compute identically reliable statistics. Setting $G = 1$ pools all channels like layer normalization, while $G = C$ normalizes each channel alone. The working middle is $G = 32$ .

Worked example: channels 2, 4, 6, 8 in two groups

One example, four channels, $G = 2$ . Group one holds $[2.0, 4.0]$ with mean $\mu = 3.0$ and variance $\sigma^2 = 1.0$ . Normalizing gives:

(2.0 - 3.0) / 1.0 = -1.0, \quad (4.0 - 3.0) / 1.0 = 1.0

Group two holds $[6.0, 8.0]$ with mean $7.0$ and variance $1.0$ , giving $-1.0$ and $1.0$ the same way. Every group leaves with mean $0.0$ and unit spread, computed without touching any other example. The diagram draws exactly this: a two-channel block inside one row becoming minus 1 and 1. Dropout pairs with this pattern for a different reason, silencing activations rather than rescaling them.

Code

The point is the check: rebuild the two-group normalization and confirm each group exits as minus 1 and 1 with overall mean zero.

1x: list[float] = [2.0, 4.0, 6.0, 8.0]2eps: float = 1e-53
4def norm(group: list[float]) -> list[float]:5    m: float = sum(group) / len(group)6    v: float = sum((t - m) ** 2 for t in group) / len(group)7    return [(t - m) / (v + eps) ** 0.5 for t in group]8
9g1: list[float] = norm(x[0:2])10g2: list[float] = norm(x[2:4])11
12print([f"{v:.4f}" for v in g1])  # -> ['-1.0000', '1.0000']13print([f"{v:.4f}" for v in g2])  # -> ['-1.0000', '1.0000']14print(f"{sum(g1 + g2) / 4:.5f}")  # -> 0.00000

Change the input to $[2.0, 2.0, 6.0, 8.0]$ and the first group collapses toward zeros: no spread in, no spread out. Normalization preserves relative gaps, never invents them.

Watch Out For

A group count that does not divide the channel count

Symptom: inserting group normalization into a layer with 10 channels and 32 groups errors, or a framework silently pads and the statistics cover padded zeros.

Fix: check every adapted layer, since widths differ across a net. Pick $G$ dividing all of them, commonly 32 only when widths allow, else the largest divisor near 32.

Replacing batch normalization where the batch is already large

Symptom: swapping a healthy large-batch classifier to groups loses a point of accuracy and trains slightly less stably, because the noisy batch statistics were acting as a mild regularizer.

Fix: keep batch normalization where batches are reliably large and inference can use running averages. Spend group normalization where the batch is small, which is the failure it was built to remove.

The Quick Version

Group normalization splits channels into G groups and normalizes each group inside one example only.
Our channels 2 and 4 form one group with mean 3 and variance 1, exiting as minus 1 and 1.
Channels 6 and 8 do the same with mean 7, so every group leaves with mean zero.
Batch size never enters the computation, so size 2 behaves like size 200.
G equals 1 recovers layer-like behavior and G equals channels recovers channel-alone behavior, with 32 the tested default.
Large-batch classifiers may still prefer batch normalization for its regularization and cheaper inference.