Batch Normalization
A technique that standardizes the activations of a neural network layer across a training batch to drastically stabilize and accelerate the learning process.
Think of It Like This
Like a microphone operator constantly adjusting the volume knob during a live speech so the speaker is never too quiet to hear or too loud to understand.
As parameters update during training, the distribution of inputs to subsequent layers constantly shifts—a problem known as internal covariate shift. Batch norm counters this by centering and scaling the outputs before passing them forward. While highly effective in CNNs, it struggles with small batch sizes and has largely been replaced by Layer Normalization in transformers.