Layer Normalization

Standardise each example from its own features alone, never from the batch around it. The same sentence normalises identically whether it arrives solo or with ninety-nine others.

Batch statistics run down a column across examples while layer statistics run across one row, so inputs 10, 12, 8 and 14 always give mean 11.0 and variance 5.0

Why Does This Exist?

Batch normalization suits image models with large fixed-size batches. A sequence model translating sentences breaks both assumptions behind it. Sentences arrive at different lengths, an 8-word one beside a 40-word one, so no shared per-position statistic lines up cleanly. Generation often runs one sequence at a time, and statistics from a batch of one are just that example wearing a disguise, with no averaging and no clean training-to-inference handoff.

The transformer needed a scheme that never looks across the batch at all. Compute mean and variance from one example own features, and the same example normalises identically in a batch of one or a batch of one hundred. That is layer normalization. The layer in the name means across one layer full feature vector for one example, not across many examples.

This page covers that per-example computation. The batch-axis twin it replaces is batch normalization, the architecture it lives in is transformers, and the sub-layer it wraps is the attention mechanism.

Think of It Like This

Curving your own exam paper

Batch normalization curves a whole exam hall. Your grade depends on how everyone else in the room did, and with one student in the room there is no curve at all, only your raw score relabelled.

Layer normalization curves your paper alone. Take every answer you wrote, standardise those scores against your own mean and your own spread, and hand back a result built from nothing but your work. Sit the exam solo or beside ninety-nine others and the curved grade comes out identical, because the room never entered the computation.

Each answer is one feature, your mean across answers is mu, your spread is sigma, and the room is the batch. The diagram draws the two readings: one band down a column of many papers, one band across a single row.

Where it stops: a student who aces everything and one who fails everything both flatten to the same shape, so the curve keeps relative pattern and discards absolute level by design.

How It Actually Works

For one example feature vector $x$ in $\mathbb{R}^d$ , meaning all $d$ features at one position:

\hat{x} = \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}}, \qquad y = \gamma \odot \hat{x} + \beta

Here $\mu$ is the mean of that example own $d$ features, $\sigma^2$ their variance, $\epsilon$ a small constant keeping division stable, and $\gamma$ and $\beta$ learned per-feature scale and shift reapplied afterwards. The formula mirrors batch norm. The entire difference is which axis $\mu$ and $\sigma^2$ are computed across: features of one example, never examples of a batch.

Worked example: one row, four features

Take the row $[10, 12, 8, 14]$ . The mean:

\mu = \frac{10 + 12 + 8 + 14}{4} = 11.0

The variance:

\sigma^2 = \frac{1 + 1 + 9 + 9}{4} = \frac{20}{4} = 5.0

With $\sigma = \sqrt{5} \approx 2.2361$ :

\hat{x} = [-0.4472, 0.4472, -1.3416, 1.3416]

Add ninety-nine wild rows beside it and nothing moves, because no term ever reads another row. Training and inference run the identical computation with no running averages to synchronise, which is the batch-norm bookkeeping that variable-length models found hardest to live with.

Where it sits in a transformer block

Every transformer block wraps its attention sub-layer and its feed-forward sub-layer in this step, historically after each sub-layer output rejoins the residual stream, with pre-norm orderings preferred in later practice. Either way the choice traces to the same cause: variable lengths plus small per-device batches, exactly where batch-axis statistics have nothing solid to stand on.

Code

The point is batch independence: one row normalised alone and inside a crowd of ninety-nine, with identical output.

1row: list[float] = [10.0, 12.0, 8.0, 14.0]2mean: float = sum(row) / len(row)3var: float = sum((x - mean) ** 2 for x in row) / len(row)4normed: list[float] = [round((x - mean) / (var + 1e-9) ** 0.5, 4) for x in row]5print(mean, var)  # -> 11.0 5.06print(normed)  # -> [-0.4472, 0.4472, -1.3416, 1.3416]

The printout matches the hand computation: mean 11.0, variance 5.0, normalised row as above. Since the formula indexes only the one row, appending other rows to the batch cannot change a single digit.

Watch Out For

Normalising across the wrong axis

Batch norm and layer norm share a formula and differ only in the axis of the mean and variance. A transposed tensor or a wrong axis argument in a hand-rolled implementation silently computes one when the other was intended, with identical output shapes so nothing crashes.

Symptom: a sequence model that converges poorly only on mixed-length batches, while fixed-length tests look fine. Fix: assert the normalised output of one example is bit-identical alone and inside a large batch before training further.

Swapping it in as an identical twin of batch norm

Layer norm is often reached for as a drop-in with the same job description batch norm once carried. Both steady activation scales, but across different axes and for different reasons: batch independence and variable lengths here, large fixed image batches there.

Symptom: an unexplained accuracy gap after swapping one for the other with all else fixed. Fix: check which axis the model needs invariance across first, and retune scale, placement, and learning rate rather than assuming a free exchange.

The Quick Version

Layer normalization computes mean and variance across one example own features, never across the batch.
Inputs 10, 12, 8 and 14 give mean 11.0 and variance 5.0, normalised to minus 0.4472, 0.4472, minus 1.3416 and 1.3416.
The same example normalises identically alone or beside ninety-nine others, with no running averages.
Variable lengths and single-example inference work because no statistic crosses examples or positions.
Transformer blocks wrap both sub-layers in it for exactly this batch independence.
Swapping batch and layer normalization without checking the axis trades one set of assumptions for another.