Comparison

MHA vs MQA vs GQA vs MLA

Comparing attention mechanisms and their impact on KV-cache memory.

MHAvsMQA / GQA / MLA

Verdict: Use GQA as the modern default to balance quality and inference cost; use MLA when working with extreme long-context regimes to drastically compress the KV cache.

Multi-Head Attention stores keys per query, while Grouped-Query Attention groups queries to share keys, vastly reducing memory overhead.

The Short Answer

The KV Cache (storing past keys and values during generation) is the primary memory bottleneck in LLM inference. Multi-Head Attention (MHA) creates independent keys/values for every query head, blowing up memory. Multi-Query Attention (MQA) forces all queries to share a single key/value pair, degrading quality. Grouped-Query Attention (GQA) strikes a balance, and Multi-Head Latent Attention (MLA) uses low-rank projections to compress it entirely.

Where They Differ

Feature	MHA	MQA	GQA	MLA
Mechanism	N queries $\rightarrow$ N keys	N queries $\rightarrow$ 1 key	N queries $\rightarrow$ G keys	N queries $\rightarrow$ 1 latent vector
KV Cache Size	Massive	Tiny	Moderate	Extremely Small
Model Quality	Highest baseline	Noticeable degradation	Almost identical to MHA	Near MHA quality via projection
Primary Use Case	Older models (GPT-3)	Edge/Fast models	Modern defaults (Llama 3)	DeepSeek-V2 / extreme contexts

Choose MHA When

You are training small models: When parameters are small ( $<1B$ ), the KV cache isn't the primary bottleneck, and giving the model maximum expressivity via full MHA yields the best results.

Choose GQA When

You are building a general-purpose LLM: Grouped-Query Attention is the industry standard today (used in Llama 2/3, Mistral, etc.). It provides 99% of the quality of MHA while reducing the KV cache footprint by a factor of 4x to 8x, allowing much larger batch sizes during serving.

Choose MLA When

You are serving 100k+ token contexts: Multi-Head Latent Attention (introduced by DeepSeek) projects the KV cache into a tiny latent space, decoding it on the fly. This allows you to serve massive context windows without dedicating hundreds of gigabytes of VRAM to the cache.

What People Get Wrong

People often think attention memory scales primarily with the parameter size of the model. In reality, KV cache memory scales strictly with the batch size × context length × number of KV heads. A 7B parameter model using MHA can easily run out of VRAM faster than a 70B parameter model using GQA if the context window is large enough.