Core ML

Group Normalization

Split a layer's channels into fixed-size groups and normalise each group within one example, never across the batch, so a detector running two images per GPU normalises exactly as reliably as one running two hundred.

Why Does This Exist?

A high-resolution object detector reading 1024×1024 medical scans fits at most two or three images per GPU before running out of memory — the images and their intermediate feature maps are simply too large to fit more. Batch normalization computes its mean and variance from the examples in a batch, and with a batch of two or three, those statistics are estimated from far too few numbers to be reliable: the same model trained with a larger effective batch would get noticeably better accuracy from that alone, purely because its normalization statistics were less noisy.

The obvious fix — layer normalization, which never reads the batch dimension at all — solves the small-batch problem but throws away something batch normalization was doing well for convolutional features: normalizing per-channel, since different channels in a CNN often detect genuinely different things (edges, textures, colors) and pooling all of them into one shared statistic per example can wash out exactly the per-channel structure a convolutional network relies on. Group normalization sits between the two: pick a fixed number of channel groups, and normalize each group within one example — batch-independent like layer normalization, but still respecting per-channel structure like batch normalization.

Think of It Like This

Grading a report card by subject group, not by student body or by every subject at once

Batch normalization grades one subject — say, math — by comparing every student in the class against each other: the curve depends on who else is in the room, and a class of three tells you almost nothing reliable about how "good" a math score is. Layer normalization grades one student by averaging across every subject on their report card at once — math, art, and gym all blended into one number — which drops the fact that math and gym measure genuinely different things.

Group normalization grades one student by splitting their report card into sensible clusters first — sciences together, arts together — and computing each cluster's own average for that student alone. No other student's grades are involved, so it works with a class of three exactly as well as a class of three hundred, and it doesn't blend math with gym into one meaningless number either.

How It Actually Works

Splitting channels into groups, then normalizing within one example

For a convolutional layer's output with $C$ channels, split those channels into $G$ groups of $C/G$ channels each. For one example, and one group, gather every value in that group — across its channels and, for a convolutional feature map, across every spatial position too — and normalize:

\hat{x}_{i} = \frac{x_i - \mu_g}{\sqrt{\sigma_g^2 + \epsilon}}, \qquad i \in \text{group } g

$\mu_g$ and $\sigma_g^2$ are the mean and variance computed from every value belonging to group $g$ , for this one example only — no other example in the batch contributes anything to $\mu_g$ or $\sigma_g^2$ . $G=1$ recovers something close to layer normalization applied per-example across all channels; $G=C$ , one channel per group, normalizes each channel entirely on its own. $G$ between those extremes, commonly 32, is where group normalization actually lives — small enough groups to keep some per-channel structure, large enough to make each group's statistics reasonably stable.

Why small-batch statistics are the real problem being solved

Estimate a mean from four random samples and the estimate carries real sampling error relative to the true population mean — measured directly, a mean computed from a batch of 4 differed from the same quantity computed across 256 examples by anywhere from 1 to 34 percent across different channels in one test run, purely from how few examples the smaller estimate drew from. Group normalization sidesteps that specific problem by never estimating anything from multiple examples at all: every statistic it computes comes from a single example's own values, so the estimate's reliability has nothing to do with batch size.

The tradeoff, honestly

Group normalization gives up batch normalization's implicit regularization effect — the noise in a small batch's statistics acts as a mild regularizer during training, which group normalization, computing exact per-example statistics, doesn't provide in the same way. It also introduces $G$ as a genuine hyperparameter to choose, unlike batch or layer normalization, which have no equivalent knob. In practice, 32 groups is a strong, well-tested default across a wide range of channel counts, but it isn't parameter-free the way the alternatives are.

Show Me the Code

Batch-of-4 statistics compared against the true population mean, then group normalization's per-example computation checked directly against zero.

1import numpy as np2
3
4def group_norm(x: np.ndarray, num_groups: int, eps: float = 1e-5) -> np.ndarray:5    n, c = x.shape6    x_g = x.reshape(n, num_groups, c // num_groups)7    mean, var = x_g.mean(axis=-1, keepdims=True), x_g.var(axis=-1, keepdims=True)8    return ((x_g - mean) / np.sqrt(var + eps)).reshape(n, c)9
10
11rng = np.random.default_rng(0)12x = rng.normal(5.0, 2.0, size=(256, 8))  # 256 examples, 8 channels13small_mean, true_mean = x[:4].mean(axis=0), x.mean(axis=0)14print(np.round(np.abs(small_mean - true_mean) / true_mean, 3))15# -> [0.029 0.129 0.019 0.016 0.339 0.225 0.134 0.007] — up to 34% off from just 4 examples16
17gn = group_norm(x[:4], num_groups=2)18print(np.round(gn.reshape(4, 2, 4).mean(axis=-1), 5))  # -> every group-example pair means ~0

A batch-normalization statistic computed from these same four examples would carry that same 1-to-34-percent noise straight into training. Group normalization's per-example groups never touch those other three examples at all, so this specific noise source doesn't apply to it.

Watch Out For

Picking a group count that doesn't divide the channel count

Group normalization requires the channel count to divide evenly by the chosen number of groups — 8 channels into 2 groups of 4 works, but 8 channels into 3 groups doesn't divide cleanly, and most implementations will either error or silently round in a way that doesn't do what was intended. When adapting an existing architecture, check the channel count at every layer where group normalization is inserted, since different layers in the same network commonly have different channel counts.

Assuming group normalization always beats batch normalization

Because group normalization solves the small-batch problem cleanly, it's tempting to use it everywhere and skip the question of what batch size is actually available. On large-batch image classification, where batch normalization's statistics are reliably estimated from hundreds of examples, batch normalization's regularizing noise and its very slightly cheaper computation at inference (a single running-average lookup rather than a fresh per-example computation) can still make it the better default. Group normalization earns its place specifically where the batch is small, not universally.

The Quick Version

Group normalization splits a layer's channels into fixed-size groups and normalizes each group using only that one example's own values.
It never reads the batch dimension, so its statistics are exactly as reliable at a batch of 2 as at a batch of 200.
Batch normalization's small-batch statistics can be off from the true population value by double-digit percentages, which is the specific failure group normalization avoids.
The number of groups, commonly 32, is a real hyperparameter — unlike batch or layer normalization, which have no equivalent choice.
It gives up batch normalization's implicit regularization from noisy small-batch statistics, which is a real tradeoff, not a free upgrade.