Skip to content
AI360XpertBeta

Grouped-Query Attention

Instead of every head keeping its own keys and values or all heads sharing one pair, grouped-query attention splits heads into a handful of groups where each group shares a KV pair, keeping most of the cache saving with little quality loss.

Four query heads share two key value groups so each group serves two heads, cutting the cache from 2.15 GB to 0.54 GB at 4096 tokens
Four query heads share two key value groups so each group serves two heads, cutting the cache from 2.15 GB to 0.54 GB at 4096 tokens

Why Does This Exist?

Serving a long-context model spends most of its memory on the KV cache: every layer stores one key and one value vector per head per token. With full multi-head attention and 32 query heads, that cache reaches 2.15 GB for a single 4096-token request, before handling any second user. Collapsing all heads to one shared pair removes the bottleneck but costs quality on tasks where heads need different views of context.

Grouped-query attention takes the middle path. Split the 32 query heads into 8 groups and give each group its own key value pair. The cache shrinks 4 fold to 0.54 GB while each group still attends differently. That trade is why production open-weight models ship with grouped heads. The mechanism it modifies is attention mechanism, its cross-sequence cousin is cross-attention, and the side-by-side verdict is MHA vs MQA vs GQA.

Think of It Like This

Four study groups sharing two libraries

Thirty-two students prepare for an exam. Giving each student a private library costs a fortune. Giving all thirty-two one shared shelf creates queues and flattens every essay into the same sources.

So the students form eight groups of four. Each group gets its own shelf, stocked slightly differently. Within a group the four students ask their own questions of the same shelf, so answers differ per student but the building only stocks eight shelves instead of thirty-two.

Where it stops: students can walk to another shelf when theirs lacks a book. Attention heads cannot. A head stuck with its group's keys attends through them or not at all, which is why the group count is fixed before training.

How It Actually Works

Queries stay per head while keys and values stay per group. With HH query heads and GG groups, each group serves H/GH/G heads. Queries keep HH projections of shape [dmodel,dhead][d_{model}, d_{head}] while keys and values keep only GG such projections each.

Worked example: 32 heads into 8 groups at 4096 tokens

Take H=32H = 32, G=8G = 8, sequence N=4096N = 4096, head width d=128d = 128, layers L=32L = 32, two bytes per value. Keys plus values per position per group per layer need 2×128×2=5122 \times 128 \times 2 = 512 bytes. Across the request:

512×8×4096×32=536,870,912 bytes=0.54 GB512 \times 8 \times 4096 \times 32 = 536{,}870{,}912 \ \mathrm{bytes} = 0.54 \ \mathrm{GB}

Full heads with G=32G = 32 give 2.152.15 GB, so 8 groups save exactly 2.15/0.54=4.02.15 / 0.54 = 4.0 fold. With G=4G = 4 the cache is 0.270.27 GB for 8.08.0 fold, and with G=1G = 1 it is 0.070.07 GB for 32.032.0 fold. The diagram draws the 4-head 2-group miniature of this arithmetic: two query heads per key value group, cache halved. The per-group computation for head hh in group gg is:

Attention(Qh,Kg,Vg)=softmax(QhKgT/d)Vg\mathrm{Attention}(Q_h, K_g, V_g) = \mathrm{softmax}\left(Q_h K_g^T / \sqrt{d}\right) V_g

Each head in the group reuses KgK_g and VgV_g but asks its own QhQ_h, so outputs still differ per head. The exact kernel that makes this fast at serving time is flash attention.

Code

The point is the check: rebuild the cache ladder from the worked numbers and confirm 2.15, 0.54, 0.27 and 0.07 GB.

seq: int = 4096d: int = 128layers: int = 32bpe: int = 2
def kv_gb(groups: int) -> float:    return 2 * groups * seq * d * layers * bpe / 1e9
mha: float = kv_gb(32)g8: float = kv_gb(8)g4: float = kv_gb(4)mqa: float = kv_gb(1)
print(f"MHA {mha:.2f}")  # -> MHA 2.15print(f"GQA8 {g8:.2f} {mha / g8:.1f}x")  # -> GQA8 0.54 4.0xprint(f"GQA4 {g4:.2f} {mha / g4:.1f}x")  # -> GQA4 0.27 8.0xprint(f"MQA {mqa:.2f} {mha / mqa:.1f}x")  # -> MQA 0.07 32.0x

Double the sequence to 8192 and every entry doubles with it. The saving ratio stays 4.0 fold because group count, not length, sets it.

Watch Out For

Treating group count as a serving-time dial

Symptom: a team tries to rerun a trained 8-group model with 4 groups to fit a busy server, and shapes mismatch at the key value projections with no clean fallback.

Fix: set GG before training and size serving for it. Group count is architecture, baked into every projection matrix, not a runtime flag.

Expecting a quick retrofit to match training with groups from scratch

Symptom: mean-pooling a trained multi-head checkpoint into groups plus brief continued training still trails a from-scratch grouped model on long-context quality.

Fix: prefer training with groups from the start for production quality. Reserve checkpoint pooling for research or tight budgets, and measure the residual gap on your own long tasks.

The Quick Version

  • Grouped-query attention gives each group of query heads one shared key value pair instead of one pair per head or one pair total.
  • With 32 heads and 8 groups, each group serves 4 heads and the KV cache shrinks 4 fold.
  • At 4096 tokens with width 128 over 32 layers, full heads need 2.15 GB while 8 groups need 0.54 GB.
  • Heads in one group share keys and values but keep their own queries, so their outputs still differ.
  • Group count is fixed at training time and cannot change at serving time.