Grouped-Query Attention
Instead of every head keeping its own keys and values or all heads sharing one pair, grouped-query attention splits heads into a handful of groups where each group shares a KV pair, keeping most of the cache saving with little quality loss.
Why Does This Exist?
Serving a long-context model spends most of its memory on the KV cache: every layer stores one key and one value vector per head per token. With full multi-head attention and 32 query heads, that cache reaches 2.15 GB for a single 4096-token request, before handling any second user. Collapsing all heads to one shared pair removes the bottleneck but costs quality on tasks where heads need different views of context.
Grouped-query attention takes the middle path. Split the 32 query heads into 8 groups and give each group its own key value pair. The cache shrinks 4 fold to 0.54 GB while each group still attends differently. That trade is why production open-weight models ship with grouped heads. The mechanism it modifies is attention mechanism, its cross-sequence cousin is cross-attention, and the side-by-side verdict is MHA vs MQA vs GQA.
Think of It Like This
Four study groups sharing two libraries
Thirty-two students prepare for an exam. Giving each student a private library costs a fortune. Giving all thirty-two one shared shelf creates queues and flattens every essay into the same sources.
So the students form eight groups of four. Each group gets its own shelf, stocked slightly differently. Within a group the four students ask their own questions of the same shelf, so answers differ per student but the building only stocks eight shelves instead of thirty-two.
Where it stops: students can walk to another shelf when theirs lacks a book. Attention heads cannot. A head stuck with its group's keys attends through them or not at all, which is why the group count is fixed before training.
How It Actually Works
Queries stay per head while keys and values stay per group. With query heads and groups, each group serves heads. Queries keep projections of shape while keys and values keep only such projections each.
Worked example: 32 heads into 8 groups at 4096 tokens
Take , , sequence , head width , layers , two bytes per value. Keys plus values per position per group per layer need bytes. Across the request:
Full heads with give GB, so 8 groups save exactly fold. With the cache is GB for fold, and with it is GB for fold. The diagram draws the 4-head 2-group miniature of this arithmetic: two query heads per key value group, cache halved. The per-group computation for head in group is:
Each head in the group reuses and but asks its own , so outputs still differ per head. The exact kernel that makes this fast at serving time is flash attention.
Code
The point is the check: rebuild the cache ladder from the worked numbers and confirm 2.15, 0.54, 0.27 and 0.07 GB.
seq: int = 4096d: int = 128layers: int = 32bpe: int = 2
def kv_gb(groups: int) -> float: return 2 * groups * seq * d * layers * bpe / 1e9
mha: float = kv_gb(32)g8: float = kv_gb(8)g4: float = kv_gb(4)mqa: float = kv_gb(1)
print(f"MHA {mha:.2f}") # -> MHA 2.15print(f"GQA8 {g8:.2f} {mha / g8:.1f}x") # -> GQA8 0.54 4.0xprint(f"GQA4 {g4:.2f} {mha / g4:.1f}x") # -> GQA4 0.27 8.0xprint(f"MQA {mqa:.2f} {mha / mqa:.1f}x") # -> MQA 0.07 32.0xDouble the sequence to 8192 and every entry doubles with it. The saving ratio stays 4.0 fold because group count, not length, sets it.
Watch Out For
Treating group count as a serving-time dial
Symptom: a team tries to rerun a trained 8-group model with 4 groups to fit a busy server, and shapes mismatch at the key value projections with no clean fallback.
Fix: set before training and size serving for it. Group count is architecture, baked into every projection matrix, not a runtime flag.
Expecting a quick retrofit to match training with groups from scratch
Symptom: mean-pooling a trained multi-head checkpoint into groups plus brief continued training still trails a from-scratch grouped model on long-context quality.
Fix: prefer training with groups from the start for production quality. Reserve checkpoint pooling for research or tight budgets, and measure the residual gap on your own long tasks.
The Quick Version
- Grouped-query attention gives each group of query heads one shared key value pair instead of one pair per head or one pair total.
- With 32 heads and 8 groups, each group serves 4 heads and the KV cache shrinks 4 fold.
- At 4096 tokens with width 128 over 32 layers, full heads need 2.15 GB while 8 groups need 0.54 GB.
- Heads in one group share keys and values but keep their own queries, so their outputs still differ.
- Group count is fixed at training time and cannot change at serving time.