Multi-Head Latent Attention
An advanced attention mechanism that drastically compresses the KV cache by projecting keys and values into a shared, lower-dimensional latent space.
Think of It Like This
Like saving a massive library of high-res photos as compressed thumbnails in RAM, only expanding them precisely when viewed.
Pioneered by models like DeepSeek-V2, MLA solves the massive memory bottleneck of standard multi-head attention during long-context inference. It achieves the representational power of full attention but with the memory footprint of grouped-query attention. This enables highly efficient serving and much larger context windows.