Skip to content
AI360XpertBeta

Context Degradation

A longer prompt doesn't make a model smarter. Past a point every extra paragraph steals attention from the answer, and facts buried in the middle go missing first.

Stuffing a prompt with filler spreads attention thin, so one answer chunk falls from about two thirds of the weight to about one fifth
Stuffing a prompt with filler spreads attention thin, so one answer chunk falls from about two thirds of the weight to about one fifth

Why Does This Exist?

A bigger window feels like a bigger desk. If the model can see 100,000 tokens at once, why not paste the whole manual in and ask your question? You save yourself the work of picking the right pages.

Because seeing is not the same as weighing. A context window sets the hard limit: text past the edge never enters the model at all. Context degradation is the soft failure inside that limit. The text fits, the model reads it, and the answer still gets worse as you add filler. Researchers call the U shaped recall curve lost in the middle: facts at the very start or the very end get found, facts buried mid prompt go missing first.

You need one fact before the mechanism makes sense. A context window is a fixed frame of tokens the model scores in one pass. This page is about why quality falls inside that frame. How you split the frame belongs to context budgeting, and fetching only the right pages belongs to retrieval-augmented generation.

Think of It Like This

One quiz slip buried in a stack of newspapers

You ask a friend to grade one quiz answer. In the short version you hand over three sheets: the answer slip plus two notes. Your friend spots the slip in seconds.

Now you bury the same slip inside twelve thick Sunday newspapers and say take your time. Your friend still checks the top paper and the bottom paper carefully, then starts skimming. An hour later you get a confident grade for the wrong slip. More paper didn't help. It gave every page a claim on limited eyes and patience, and the middle pages paid for it.

It stops here. Your friend can re read the stack, ask you which section matters, or throw half of it away. A model gets one forward pass with fixed weights, and its middle blindness comes from scores and positions, not tired eyes. The shape of the failure matches. The cause does not.

How It Actually Works

For each token it writes, the model scores every token you gave it, turns those scores into weights that sum to 1, and blends what it read by those weights. Filler does not sit quietly on the side. It joins the vote and takes weight away from the answer.

Attention is a fixed budget

Call the number of chunks you hand over nn, in plain words how many pieces compete. Call the relevance score of chunk ii by the name sis_i, in plain words how strongly the model thinks chunk ii matches the current question. Call its final share wiw_i, in plain words the fraction of attention chunk ii receives. The rule that turns scores into shares is the softmax:

wi=exp⁡(si)∑j=1nexp⁡(sj)w_i = \frac{\exp(s_i)}{\sum_{j=1}^{n} \exp(s_j)}

Here exp⁡\exp is the exponential function and the bottom sum runs over all nn chunks. The shares always add to 1. That's the whole trap. When nn grows, the bottom sum grows, so a fixed score sis_i buys a smaller share wiw_i. You didn't make the answer worse. You gave it more rivals.

Position tilts what is left

Dilution alone would hurt every position equally. In practice the middle hurts first. Training data teaches the pattern: instructions sit at the start, answers and summaries sit at the end, and the filler lives between them. Positional signals and causal masking then favor the edges, so the model behaves like your friend with the newspapers. It checks the top sheet and the bottom sheet, then skims. Put the key sentence at 50 percent depth and it competes on worse terms than the same sentence pasted last.

Reasoning breaks before copy paste retrieval does. Finding one password string is a single match. Answering from three facts on three far apart pages needs all three shares to survive at once, and one thin share sinks the chain. That's why a demo that asks for one quoted line can look fine while the real task, which compares and combines, falls apart in the same prompt.

Worked example: from 0.665 to 0.208

Take three chunks with scores [2,1,0][2, 1, 0]. In words: the answer chunk scores 2, a near miss scores 1, one filler scores 0. Exponentiate: exp⁡(2)≈7.389\exp(2) \approx 7.389, exp⁡(1)≈2.718\exp(1) \approx 2.718, exp⁡(0)=1.000\exp(0) = 1.000. Add them: 7.389+2.718+1.000=11.1077.389 + 2.718 + 1.000 = 11.107. Divide:

w=[7.389/11.107, 2.718/11.107, 1.000/11.107]≈[0.665, 0.245, 0.090]w = [7.389 / 11.107,\ 2.718 / 11.107,\ 1.000 / 11.107] \approx [0.665,\ 0.245,\ 0.090]

The answer holds about two thirds of the weight. Now keep the same three scores and add nine more filler chunks, each scoring 1. Each adds about 2.7182.718, so the new bottom sum is 11.107+9×2.718=11.107+24.462=35.56911.107 + 9 \times 2.718 = 11.107 + 24.462 = 35.569. The answer score didn't move, but its share is now 7.389/35.569≈0.2087.389 / 35.569 \approx 0.208. Same answer, same match, about one fifth of the say. The diagram above draws exactly these two bars.

Name the trade off plainly. Stuffing the window saves you pipeline work: no chunking, no ranking, no ordering choices. It costs recall first, then latency and price on every request, since each extra token is scored. Don't use a giant prompt when the answer lives in a small slice of a large store. That's the retrieval case, and tight top chunks plus a short prompt win. The method assumes every position is heard equally once inside the window. That assumption fails because the softmax budget plus edge favoring position signals make the middle quiet.

Code

This snippet replays the worked example so you can feel the dilution. It builds one short score list and one long one, runs a plain softmax, and prints the answer share in each case.

import math

def softmax(scores: list[float]) -> list[float]:    m = max(scores)    exps = [math.exp(s - m) for s in scores]    total = sum(exps)    return [e / total for e in exps]

short_scores: list[float] = [2.0, 1.0, 0.0]long_scores: list[float] = [2.0, 1.0, 0.0] + [1.0] * 9
short_w = softmax(short_scores)long_w = softmax(long_scores)
print(f"{short_w[0]:.3f}")  # -> 0.665print(f"{long_w[0]:.3f}")  # -> 0.208print(f"{sum(long_w):.3f}")  # -> 1.000

The answer chunk keeps its score of 2.0 in both lists, yet its printed share drops from 0.665 to 0.208. The third print is a sanity check: the shares still sum to 1.000, so nothing vanished. It moved to the filler.

Watch Out For

Raising top_k to be safe

You set retrieval to return twenty chunks instead of four because missing the answer feels scarier than adding noise. Recall of the retriever goes up a touch. Answer quality goes down a lot, and the failure is quiet: fluent sentences, confident tone, wrong details, citations that point near but not at the claim.

Keep the prompt tight instead. Retrieve a small set, rerank it, and hand the model three to five strong chunks with the best one placed last. If you must pass more, test whether each added chunk moves a graded answer score before you ship it.

Testing with the needle pasted at the end

Your check buries one password line at the very end of the prompt, the model quotes it back, and you declare long context solved. Production then asks a question whose evidence sits mid document and needs two facts combined, and it fails on the same prompt length that passed your test.

Test the shape you will ship. Place the key facts at 0, 25, 50, 75, and 100 percent depth, ask questions that combine facts rather than repeat one line, and grade the answers. If the middle depths sag, your prompt is too long no matter what the end depth scored.

The Quick Version

  • Context degradation is quality loss inside the window: the text fits, yet the answer gets worse as filler grows.
  • Attention shares sum to 1, so every added chunk takes weight from the answer even when the answer score never changes.
  • Middle positions fail first, which is why the same fact is found at the start or end and missed mid prompt.
  • Multi step reasoning over scattered facts breaks before single fact lookup does.
  • Tight retrieval with a few ranked chunks beats pasting the whole store when the answer lives in a small slice.