Context Windows

Why Does This Exist?

A transformer never reads your whole document. It reads a fixed run of tokens, at most $n_{\max}$ of them, in a single pass. That ceiling is set when the model is trained, and no clever prompting stretches it at request time. This page is about that frame: the context window.

You might wonder why the ceiling exists at all. Self-attention compares every token against every other token, so a pass over $n$ tokens does work that grows with $n^2$, not $n$. The KV cache held during generation grows with $n$ too. Without a fixed cap, one long request could demand an unbounded amount of compute and memory, so every model ships with the cap baked in.

Two neighbours handle what this page doesn't. Long-context extension asks how a trained window gets widened after the fact. Context degradation asks why recall gets uneven inside very long windows. Here you get the frame itself: what counts as inside, what slides out, and what bigger frames charge.

The one idea you need first: models read tokens, not words. Tokenization chops your text into those pieces, and those pieces outnumber your words, so count tokens against the limit, never words.

Think of It Like This

How It Actually Works

Call the number of tokens in one pass $n$, where $n$ is a plain count ($n = 8$ means eight tokens). Call the trained ceiling $n_{\max}$, the largest $n$ the model ever saw in training. The window is the frame that holds at most $n_{\max}$ tokens at a time.

The frame

Membership is binary. A token with position index $i$, where $i$ counts from the start of the pass, is either inside the frame and visible to attention, or past an edge and simply not input. There is no half-visible state. Positional encoding tells the model where each inside token sits, but only positions up to $n_{\max}$ carry meanings the model learned, which is why text past the edge can't just be squeezed in.

When the frame is full and generation continues, it slides: each fresh token takes the newest slot and the oldest token leaves. During a long chat this is silent. Nothing warns you. The start of the conversation just stops being input.

Why the frame can't just grow

Dense self-attention forms one query per token and scores it against every key, which is $n \times n = n^2$ ordered pairs. At $n = 8$ that's 64 comparisons. At $n = 16$ it's 256. Twice the length, four times the pairs. The KV cache adds its own bill: every extra token stores its key and value vectors in every layer for as long as generation runs, so memory climbs with $n$ alongside the compute.

So a bigger window buys one thing, recall of more distant context, and sells two, latency and memory. That's the trade it always charges. When only a handful of passages actually matter, stuffing the whole library into the frame wastes both, and retrieving the passages instead wins.

Worked example

Fix the frame at $W = 8$ slots, where $W$ is just the worked size for this example. Your prompt fills it exactly:

$[t_1, t_2, t_3, t_4, t_5, t_6, t_7, t_8], \quad n = 8, \quad 8^2 = 64 \text{ ordered pairs}$

You generate $t_9$. The frame slides:

$[t_2, t_3, t_4, t_5, t_6, t_7, t_8, t_9]$

Token $t_1$ isn't deprioritized. It's gone.

You generate $t_{10}$:

$[t_3, t_4, t_5, t_6, t_7, t_8, t_9, t_{10}]$

Same length, same 64 pairs. The diagram above draws exactly this state: $t_1$ and $t_2$ off the left edge, $t_{11}$ and $t_{12}$ waiting on the right.

Now compare a frame twice as wide. At $n = 16$ the pairs are $16^2 = 256$, and $256 / 64 = 4$. Twice the tokens, four times the comparisons. That ratio is why context length is a headline specification instead of a footnote.

The assumption underneath: the material you need fits inside the recent $n_{\max}$ tokens. When it doesn't, the slide drops whatever is oldest, however important it was.

Code

The point of this snippet is the eviction rule, not the arithmetic. Watch the oldest tokens leave and the length refuse to grow.

1def slide_window(tokens: list[int], incoming: list[int], window: int) -> list[int]:2    """Keep only the newest `window` tokens; oldest leave first."""3    kept = tokens + incoming4    return kept[-window:]5
6
7def attention_pairs(n: int) -> int:8    """Ordered query-key comparisons in dense self-attention."""9    return n * n10
11
12window = list(range(1, 9))  # [t1 .. t8], the full 8-slot frame13print(window)  # -> [1, 2, 3, 4, 5, 6, 7, 8]14window = slide_window(window, incoming=[9, 10], window=8)15print(window)  # -> [3, 4, 5, 6, 7, 8, 9, 10]16print(attention_pairs(8))  # -> 6417print(attention_pairs(16))  # -> 256

Three things to notice. The length never exceeds 8. The survivors keep their order, $t_3$ first and $t_{10}$ last. And the pair counts, 64 and 256, match the hand computation above, which is the whole point of printing them.

Watch Out For

The Quick Version

• A context window is the fixed number of tokens a model reads in one pass, capped at training time. Nothing outside it reaches the model.
• When the window is full, each new token evicts the oldest one first. The length never exceeds the cap.
• Attention compares every token pair, so doubling the length roughly quadruples the comparison work. Eight tokens need 64 pairs and sixteen need 256.
• Cached keys and values grow with length too, so long windows tax memory as well as compute.
• Fitting inside the limit isn't the same as being used well. Placement inside the window still affects recall.
• When only a few passages matter, retrieve them instead of widening the window.

What to Read Next

• Tokenization
• KV Cache
• Long-Context Extension
• Attention Sinks and Streaming