Sliding Window Attention
Each token only attends to the nearest W tokens around it instead of every token in the sequence, so attention cost scales linearly with sequence length rather than quadratically.
Why Does This Exist?
You want a model to read a 32,000 token contract on one GPU. Full self-attention compares every token with every other token, so doubling length quadruples work. At 32K that is over a billion score cells in one layer, which does not fit.
The fix starts from a plain fact about language. Most of what predicts the next word sits nearby: the current sentence, the open bracket a few lines up, the variable assigned last paragraph. Distant tokens matter, but only a few of them. A window that keeps the dense local part and lets depth carry the rest matches that shape.
One idea underneath it. A score cell is one query meeting one key. Full attention fills an N by N grid. A causal window of width W keeps at most W cells per row, so the grid holds about N times W cells. That is the whole saving, and the diagram above draws it for six tokens.
This page covers the window, the receptive field through depth, and the limit. The exact attention operation lives in self-attention. The memory efficient exact rewrite lives in flash-attention. The cross sequence variant lives in cross-attention.
Think of It Like This
Reading with a three page slider
You read a long report through a cardboard slider that shows three pages at a time. You slide it forward one page at a time and write a margin note at each stop from what you can see. Each note only ever saw three pages, yet a note on page 50 was written by someone who had already read the notes from pages 47 to 49, which themselves saw earlier pages.
The slider is the window W. Each margin note is one layer representation. Sliding the slider is the causal constraint. Stacking notes through layers is how reach grows beyond one window.
Where it stops: your notes never teleport. If page 50 truly needs a number from page 2 and no intermediate note carried it forward, the slider misses it. That is when you need a global token or retrieval, not a wider slider alone.
How It Actually Works
Call the sequence length N and the window width W. Query at position i may score keys from through in a causal model. Every other cell is masked and costs nothing. Per layer cost falls from to . Here means grows with, is length, is the fixed window.
Depth restores reach. One layer sees W back. Two layers see about back, because the second layer reads first layer states that already summarise their own windows. With L layers the receptive field is about tokens. A 32 layer model with reaches about tokens back through composition, which covers a 32K context without any single layer scanning that far.
Worked example with six tokens
Take and with a causal window. Row counts are exact. Row 0 sees 1 key. Row 1 sees 2. Rows 2 through 5 see 3 each. Total kept cells are . Full attention would compute . You save cells, about 58 percent, for a ratio of . That is the right hand grid in the diagram: 15 mint cells, the rest blank.
What the window cannot do alone
A pure window cannot link two tokens apart in one layer. If the answer on the last page depends on a code on the first page and nothing in between repeats it, intermediate layers may dilute it before it arrives. The standard repair keeps a few global tokens that attend everywhere and are attended everywhere, at for such tokens with far smaller than . You pay a little to keep exact long jumps.
Code
Count score cells for full versus causal window attention with plain Python. No library needed.
def full_cells(n: int) -> int: return n * n
def window_cells(n: int, w: int) -> int: return sum(min(w, i + 1) for i in range(n))
for n in (4096, 8192, 32768): f = full_cells(n) loc = window_cells(n, 4096) print(f"n={n} full={f:,} window={loc:,} ratio={f/loc:.1f}x")# -> n=4096 full=16,777,216 window=8,390,656 ratio=2.0x# -> n=8192 full=67,108,864 window=25,167,872 ratio=2.7x# -> n=32768 full=1,073,741,824 window=125,831,168 ratio=8.5xAt 32K with the window needs about 126 million cells against about 1.07 billion for full attention. The saving grows with N, which is exactly when you need it.
Watch Out For
Expecting the KV cache to shrink with the window
You set on a 32K run and watch memory stay high. The window limits which cached keys get scored, not which get stored. Decoding still keeps all past keys and values unless you add explicit eviction.
Symptom: memory grows with N despite the window, and profiling shows cache near full length. Fix: keep the window for compute and add a separate eviction policy for memory, or pick a method that documents cache eviction. Do not budget memory from the compute figure.
Trusting one window for a needle in a long document
A test asks for a code from page 2 while reading page 200. Pure windows pass information through dozens of compositions, and the exact string can smear before it arrives.
Symptom: near perfect local fluency with misses on planted long jumps that full attention gets right. Fix: add a few global tokens, a retrieval step, or a wider window at one or two layers, then retest the planted jump. Match the mechanism to the locality your task actually has.
The Quick Version
- Sliding window attention lets query i score about W past keys instead of all N, so per layer cost is not .
- Our six token example with width three keeps 15 cells out of 36, a 2.4x saving you can count in the diagram.
- L layers reach about back through composition, so 32 layers of width 4096 cover a 32K context.
- A pure window cannot make a one hop long jump, so add a few global tokens at when exact recall matters.
- The window trims compute, not cache memory, unless eviction is added separately.