QK-Normalisation
A stabilization technique in transformers where Layer Normalization is applied directly to the Query and Key vectors before calculating attention scores.
Think of It Like This
Like putting sunglasses on two incredibly bright spotlights so you can actually measure their overlap without completely blinding the camera.
In large-scale models, attention logits can easily explode, causing the softmax function to become extremely sharp and destabilizing training. QK-Norm prevents these vectors from growing too large in magnitude. This is particularly crucial for training massive architectures or extremely deep networks where numerical instability is rampant.