Grouped-Query Attention
An optimized attention mechanism that speeds up inference by grouping query heads to share a single key and value head, reducing memory bandwidth usage.
Think of It Like This
Like having a team of researchers all share a single master filing cabinet of reference documents rather than printing individual copies for every person.
Grouped-Query Attention (GQA) strikes a balance between Multi-Head Attention (MHA) and Multi-Query Attention (MQA). By sharing key-value projections across a small group of query heads, it significantly reduces the size of the KV cache during text generation. This allows large language models to process long contexts with high speed and minimal performance degradation.