RMSNorm
A normalization layer that rescales a vector by its own root-mean-square magnitude alone, skipping the mean-centering step layer normalization performs.
Divide a vector by the root-mean-square of its own entries, then apply a learned scale — no mean subtraction, no learned shift. Dropping the mean-centering step removes roughly a third of layer normalization's arithmetic and one of its two learned parameters, while matching its accuracy closely enough that most modern large language models ship it instead.
The justification is that re-scaling, not re-centering, is what actually stabilizes training in practice for these architectures; the mean-subtraction step layer normalization performs turns out to matter far less than its variance-control half.