Context Degradation
A longer prompt doesn't make a model smarter. Past a point every extra paragraph steals attention from the answer, and facts buried in the middle go missing first.
Why Does This Exist?
A bigger window feels like a bigger desk. If the model can see 100,000 tokens at once, why not paste the whole manual in and ask your question? You save yourself the work of picking the right pages.
Because seeing is not the same as weighing. A context window sets the hard limit: text past the edge never enters the model at all. Context degradation is the soft failure inside that limit. The text fits, the model reads it, and the answer still gets worse as you add filler. Researchers call the U shaped recall curve lost in the middle: facts at the very start or the very end get found, facts buried mid prompt go missing first.
You need one fact before the mechanism makes sense. A context window is a fixed frame of tokens the model scores in one pass. This page is about why quality falls inside that frame. How you split the frame belongs to context budgeting, and fetching only the right pages belongs to retrieval-augmented generation.
Think of It Like This
One quiz slip buried in a stack of newspapers
You ask a friend to grade one quiz answer. In the short version you hand over three sheets: the answer slip plus two notes. Your friend spots the slip in seconds.
Now you bury the same slip inside twelve thick Sunday newspapers and say take your time. Your friend still checks the top paper and the bottom paper carefully, then starts skimming. An hour later you get a confident grade for the wrong slip. More paper didn't help. It gave every page a claim on limited eyes and patience, and the middle pages paid for it.
It stops here. Your friend can re read the stack, ask you which section matters, or throw half of it away. A model gets one forward pass with fixed weights, and its middle blindness comes from scores and positions, not tired eyes. The shape of the failure matches. The cause does not.
How It Actually Works
For each token it writes, the model scores every token you gave it, turns those scores into weights that sum to 1, and blends what it read by those weights. Filler does not sit quietly on the side. It joins the vote and takes weight away from the answer.
Attention is a fixed budget
Call the number of chunks you hand over , in plain words how many pieces compete. Call the relevance score of chunk by the name , in plain words how strongly the model thinks chunk matches the current question. Call its final share , in plain words the fraction of attention chunk receives. The rule that turns scores into shares is the softmax:
Here is the exponential function and the bottom sum runs over all chunks. The shares always add to 1. That's the whole trap. When grows, the bottom sum grows, so a fixed score buys a smaller share . You didn't make the answer worse. You gave it more rivals.
Position tilts what is left
Dilution alone would hurt every position equally. In practice the middle hurts first. Training data teaches the pattern: instructions sit at the start, answers and summaries sit at the end, and the filler lives between them. Positional signals and causal masking then favor the edges, so the model behaves like your friend with the newspapers. It checks the top sheet and the bottom sheet, then skims. Put the key sentence at 50 percent depth and it competes on worse terms than the same sentence pasted last.
Reasoning breaks before copy paste retrieval does. Finding one password string is a single match. Answering from three facts on three far apart pages needs all three shares to survive at once, and one thin share sinks the chain. That's why a demo that asks for one quoted line can look fine while the real task, which compares and combines, falls apart in the same prompt.
Worked example: from 0.665 to 0.208
Take three chunks with scores . In words: the answer chunk scores 2, a near miss scores 1, one filler scores 0. Exponentiate: , , . Add them: . Divide:
The answer holds about two thirds of the weight. Now keep the same three scores and add nine more filler chunks, each scoring 1. Each adds about , so the new bottom sum is . The answer score didn't move, but its share is now . Same answer, same match, about one fifth of the say. The diagram above draws exactly these two bars.
Name the trade off plainly. Stuffing the window saves you pipeline work: no chunking, no ranking, no ordering choices. It costs recall first, then latency and price on every request, since each extra token is scored. Don't use a giant prompt when the answer lives in a small slice of a large store. That's the retrieval case, and tight top chunks plus a short prompt win. The method assumes every position is heard equally once inside the window. That assumption fails because the softmax budget plus edge favoring position signals make the middle quiet.
Code
This snippet replays the worked example so you can feel the dilution. It builds one short score list and one long one, runs a plain softmax, and prints the answer share in each case.
import math
def softmax(scores: list[float]) -> list[float]: m = max(scores) exps = [math.exp(s - m) for s in scores] total = sum(exps) return [e / total for e in exps]
short_scores: list[float] = [2.0, 1.0, 0.0]long_scores: list[float] = [2.0, 1.0, 0.0] + [1.0] * 9
short_w = softmax(short_scores)long_w = softmax(long_scores)
print(f"{short_w[0]:.3f}") # -> 0.665print(f"{long_w[0]:.3f}") # -> 0.208print(f"{sum(long_w):.3f}") # -> 1.000The answer chunk keeps its score of 2.0 in both lists, yet its printed share drops from 0.665 to 0.208. The third print is a sanity check: the shares still sum to 1.000, so nothing vanished. It moved to the filler.
Watch Out For
Raising top_k to be safe
You set retrieval to return twenty chunks instead of four because missing the answer feels scarier than adding noise. Recall of the retriever goes up a touch. Answer quality goes down a lot, and the failure is quiet: fluent sentences, confident tone, wrong details, citations that point near but not at the claim.
Keep the prompt tight instead. Retrieve a small set, rerank it, and hand the model three to five strong chunks with the best one placed last. If you must pass more, test whether each added chunk moves a graded answer score before you ship it.
Testing with the needle pasted at the end
Your check buries one password line at the very end of the prompt, the model quotes it back, and you declare long context solved. Production then asks a question whose evidence sits mid document and needs two facts combined, and it fails on the same prompt length that passed your test.
Test the shape you will ship. Place the key facts at 0, 25, 50, 75, and 100 percent depth, ask questions that combine facts rather than repeat one line, and grade the answers. If the middle depths sag, your prompt is too long no matter what the end depth scored.
The Quick Version
- Context degradation is quality loss inside the window: the text fits, yet the answer gets worse as filler grows.
- Attention shares sum to 1, so every added chunk takes weight from the answer even when the answer score never changes.
- Middle positions fail first, which is why the same fact is found at the start or end and missed mid prompt.
- Multi step reasoning over scattered facts breaks before single fact lookup does.
- Tight retrieval with a few ranked chunks beats pasting the whole store when the answer lives in a small slice.