FlashAttention
Standard attention writes a huge score matrix to slow GPU memory and reads it back. FlashAttention works in small tiles that fit in fast on chip memory, so the result matches exactly with far less traffic.
Why Does This Exist?
Attention mechanism builds an by score matrix. For that is scores. At four bytes each that is MB for one head and GB for heads in one layer. Standard code writes that matrix to slow HBM memory, reads it for softmax, writes weights, and reads again for the value multiply.
Modern GPUs are starved by that traffic, not by arithmetic. The fix keeps tiles of queries, keys, and values in small fast SRAM, runs the softmax piece by piece, and never stores the full matrix. FLOPs stay the same. Bytes moved collapse. The output matches to rounding.
This page covers the tiling trick. The score and blend it speeds up live in attention mechanism. Other ways to cut the same cost live in grouped-query attention and linear attention.
Think of It Like This
Settling a card league without copying every score sheet
A club runs a round robin with players. The naive way copies all pair scores onto one giant sheet in the back room, walks it out for averages, walks it back for prizes, then walks it out again to check ties.
The tiled way keeps one small clipboard at the table. Score one group, update running totals and the running best, put the group back, take the next group. The giant sheet never exists, but the final totals match exactly.
Each pair score is one attention score, the back room is HBM, the clipboard is SRAM, running totals are the online softmax state, and prizes are the output rows.
Where it stops: cards can be rescored from memory for free. Attention tiles must reload keys and values from HBM for the backward pass or recompute them, which trades extra reads for the saved write.
How It Actually Works
Call , , the query, key, and value matrices with rows and columns. Call the score matrix. Standard attention writes , reads it for row softmax, then multiplies by . That is four large HBM trips plus storage.
Tiling picks block sizes by that fit in SRAM. For each query block :
- Load one and block into SRAM.
- Form tile scores only in SRAM.
- Update a running row maximum and running normalizer with the online recurrence, which folds the new tile into the softmax without seeing all scores.
- Weight by the partial softmax and add into running output .
- After the last block, divide by final and write only back to HBM.
The by matrix never lands in HBM. Memory falls from for scores to for outputs plus tiny per row statistics. Compute stays . The backward pass stores only and per row and rebuilds tiles, which is why causal masking still applies cleanly inside tiles.
Worked example: how large the avoided write is
Take with heads and four byte floats:
That GB is the diagram headline: one wide HBM box holding , , plus a dashed missing matrix, one small SRAM box holding a single tile plus running and , and one arrow carrying only finished outputs back.
Worked example: one row tiled without changing the answer
Take one row with scores . Full softmax exponentiates to , sums to , and divides to .
Tiled with width one, start with . Running maximum and normalizer . Bring in . New maximum stays . Rescale old mass by and add . New . Divide and the same appears. Order changed, value did not.
Code
Confirm the GB figure and the tiled row match with plain Python.
import math
n: int = 4096heads: int = 32entries: int = n * nper_head_mb: float = entries * 4 / 1024 / 1024per_layer_gb: float = per_head_mb * heads / 1024
print(f"{entries} {per_head_mb:.0f} {per_layer_gb:.0f}") # -> 16777216 64 2
scores: list[float] = [2.0, 1.0]exps: list[float] = [math.exp(s) for s in scores]total: float = sum(exps)full: list[float] = [e / total for e in exps]
m: float = scores[0]l: float = 1.0for s in scores[1:]: m_new: float = m if m > s else s l = l * math.exp(m - m_new) + math.exp(s - m_new) m = m_new
tiled_first: float = math.exp(scores[0] - m) / lprint(f"{full[0]:.3f} {tiled_first:.3f}") # -> 0.731 0.731The printout matches the hand arithmetic: entries, MB per head, GB per layer, and both ways.
Watch Out For
Calling it an approximation
A new name plus faster runtime reads as a tradeoff. This one is exact up to floating point order.
Symptom: a test allows loose tolerance because the method is assumed lossy. Fix: compare against standard attention with tight tolerance near as above, and treat any larger gap as a bug in masking or scaling, not as expected loss.
Expecting fewer FLOPs
Memory falls from to but arithmetic stays . The win is bytes moved.
Symptom: no speedup on hardware where arithmetic already dominates or tiles fit poorly. Fix: profile HBM traffic first, then consider linear attention when pure operation count must fall.
The Quick Version
- Standard attention at length 4096 builds 16,777,216 scores per head in our example.
- Those scores cost about 64 MB per head and about 2 GB for 32 heads in one layer.
- FlashAttention loads small query, key, and value tiles into fast SRAM and never stores the full matrix.
- A running maximum and normalizer fold each tile into the same softmax the full matrix would give.
- Memory drops from order N squared to order N while operation count stays the same.
- The backward pass rebuilds tiles from stored row statistics instead of stored weights.