Skip to content
AI360XpertBeta

FlashAttention

Standard attention writes a huge score matrix to slow GPU memory and reads it back. FlashAttention works in small tiles that fit in fast on chip memory, so the result matches exactly with far less traffic.

A 4096 long run would write 16.7 million scores using 64 MB per head, so tiles stay in SRAM and accumulate the same output without that write
A 4096 long run would write 16.7 million scores using 64 MB per head, so tiles stay in SRAM and accumulate the same output without that write

Why Does This Exist?

Attention mechanism builds an NN by NN score matrix. For N=4096N = 4096 that is 4096×4096=16,777,2164096 \times 4096 = 16,777,216 scores. At four bytes each that is 6464 MB for one head and 22 GB for 3232 heads in one layer. Standard code writes that matrix to slow HBM memory, reads it for softmax, writes weights, and reads again for the value multiply.

Modern GPUs are starved by that traffic, not by arithmetic. The fix keeps tiles of queries, keys, and values in small fast SRAM, runs the softmax piece by piece, and never stores the full matrix. FLOPs stay the same. Bytes moved collapse. The output matches to rounding.

This page covers the tiling trick. The score and blend it speeds up live in attention mechanism. Other ways to cut the same cost live in grouped-query attention and linear attention.

Think of It Like This

Settling a card league without copying every score sheet

A club runs a round robin with 100100 players. The naive way copies all 10,00010,000 pair scores onto one giant sheet in the back room, walks it out for averages, walks it back for prizes, then walks it out again to check ties.

The tiled way keeps one small clipboard at the table. Score one group, update running totals and the running best, put the group back, take the next group. The giant sheet never exists, but the final totals match exactly.

Each pair score is one attention score, the back room is HBM, the clipboard is SRAM, running totals are the online softmax state, and prizes are the output rows.

Where it stops: cards can be rescored from memory for free. Attention tiles must reload keys and values from HBM for the backward pass or recompute them, which trades extra reads for the saved write.

How It Actually Works

Call QQ, KK, VV the query, key, and value matrices with NN rows and dd columns. Call S=QK⊤/dS = Q K^\top / \sqrt{d} the score matrix. Standard attention writes SS, reads it for row softmax, then multiplies by VV. That is four large HBM trips plus O(N2)O(N^2) storage.

Tiling picks block sizes BrB_r by BcB_c that fit in SRAM. For each query block QiQ_i:

  1. Load one KjK_j and VjV_j block into SRAM.
  2. Form tile scores Sij=QiKj⊤/dS_{ij} = Q_i K_j^\top / \sqrt{d} only in SRAM.
  3. Update a running row maximum mm and running normalizer ll with the online recurrence, which folds the new tile into the softmax without seeing all scores.
  4. Weight VjV_j by the partial softmax and add into running output OiO_i.
  5. After the last block, divide OiO_i by final ll and write only OiO_i back to HBM.

The NN by NN matrix never lands in HBM. Memory falls from O(N2)O(N^2) for scores to O(N)O(N) for outputs plus tiny per row statistics. Compute stays O(N2d)O(N^2 d). The backward pass stores only mm and ll per row and rebuilds tiles, which is why causal masking still applies cleanly inside tiles.

Worked example: how large the avoided write is

Take N=4096N = 4096 with 3232 heads and four byte floats:

4096×4096=16,777,216 scores4096 \times 4096 = 16,777,216 \text{ scores} 16,777,216×4=67,108,864 bytes ≈64 MB per head16,777,216 \times 4 = 67,108,864 \text{ bytes } \approx 64 \text{ MB per head} 64 MB ×32=2048 MB ≈2 GB per layer64 \text{ MB } \times 32 = 2048 \text{ MB } \approx 2 \text{ GB per layer}

That 22 GB is the diagram headline: one wide HBM box holding QQ, KK, VV plus a dashed missing matrix, one small SRAM box holding a single tile plus running mm and ll, and one arrow carrying only finished outputs back.

Worked example: one row tiled without changing the answer

Take one row with scores [2.0,1.0][2.0, 1.0]. Full softmax exponentiates to [7.39,2.72][7.39, 2.72], sums to 10.1110.11, and divides to [0.731,0.269][0.731, 0.269].

Tiled with width one, start with 2.02.0. Running maximum m=2.0m = 2.0 and normalizer l=1.0l = 1.0. Bring in 1.01.0. New maximum stays 2.02.0. Rescale old mass by exp⁡(2.0−2.0)=1.0\exp(2.0 - 2.0) = 1.0 and add exp⁡(1.0−2.0)≈0.368\exp(1.0 - 2.0) \approx 0.368. New l≈1.368l \approx 1.368. Divide and the same [0.731,0.269][0.731, 0.269] appears. Order changed, value did not.

Code

Confirm the 22 GB figure and the tiled row match with plain Python.

import math
n: int = 4096heads: int = 32entries: int = n * nper_head_mb: float = entries * 4 / 1024 / 1024per_layer_gb: float = per_head_mb * heads / 1024
print(f"{entries} {per_head_mb:.0f} {per_layer_gb:.0f}")  # -> 16777216 64 2
scores: list[float] = [2.0, 1.0]exps: list[float] = [math.exp(s) for s in scores]total: float = sum(exps)full: list[float] = [e / total for e in exps]
m: float = scores[0]l: float = 1.0for s in scores[1:]:    m_new: float = m if m > s else s    l = l * math.exp(m - m_new) + math.exp(s - m_new)    m = m_new
tiled_first: float = math.exp(scores[0] - m) / lprint(f"{full[0]:.3f} {tiled_first:.3f}")  # -> 0.731 0.731

The printout matches the hand arithmetic: 16,777,21616,777,216 entries, 6464 MB per head, 22 GB per layer, and 0.7310.731 both ways.

Watch Out For

Calling it an approximation

A new name plus faster runtime reads as a tradeoff. This one is exact up to floating point order.

Symptom: a test allows loose tolerance because the method is assumed lossy. Fix: compare against standard attention with tight tolerance near 1e−51e-5 as above, and treat any larger gap as a bug in masking or scaling, not as expected loss.

Expecting fewer FLOPs

Memory falls from O(N2)O(N^2) to O(N)O(N) but arithmetic stays O(N2d)O(N^2 d). The win is bytes moved.

Symptom: no speedup on hardware where arithmetic already dominates or tiles fit poorly. Fix: profile HBM traffic first, then consider linear attention when pure operation count must fall.

The Quick Version

  • Standard attention at length 4096 builds 16,777,216 scores per head in our example.
  • Those scores cost about 64 MB per head and about 2 GB for 32 heads in one layer.
  • FlashAttention loads small query, key, and value tiles into fast SRAM and never stores the full matrix.
  • A running maximum and normalizer fold each tile into the same softmax the full matrix would give.
  • Memory drops from order N squared to order N while operation count stays the same.
  • The backward pass rebuilds tiles from stored row statistics instead of stored weights.