Layer Normalization

The Problem

Batch Normalization (2015) was a massive success for convolutional networks, but it had a fatal flaw: it relied on the mini-batch. For Recurrent Neural Networks (RNNs) and natural language processing tasks, sequences vary in length. Computing batch statistics across varying lengths was complex and unstable. Furthermore, if a batch size was too small (e.g., memory limits), Batch Norm's statistics became wildly inaccurate.

The Idea

Instead of normalizing a specific feature across a whole batch of different examples, why not normalize all the features of a single example? Because this only relies on the data within one specific sample, it is entirely independent of the batch size and the length of the sequence.

How It Works

In Layer Normalization (LayerNorm), the mean and variance are calculated across the hidden units (features) in the same layer for a single data point.

For example, if a token in a sentence is represented by a 512-dimensional vector, LayerNorm calculates the mean and variance of those 512 numbers. It subtracts the mean, divides by the standard deviation, and then applies learnable scale and shift parameters. This happens independently for every token in every sequence, regardless of how many sequences are in the batch.

Why It Mattered

LayerNorm solved the normalization problem for sequence models. It made RNNs and LSTMs much faster and more stable to train. Because it behaves exactly the same during training and inference (unlike BatchNorm, which switches to running averages during inference), it is vastly easier to implement and reason about.

What Came After

Layer Normalization became the absolute standard for Natural Language Processing. When the Transformer was invented in 2017, it utilized LayerNorm inside every block. Every modern Large Language Model (GPT, Llama, Claude) relies on LayerNorm (or its simplified cousin, RMSNorm) to maintain stability across thousands of layers.