Skip to content
AI360XpertBeta

Context Compaction

Retrieval hands you far more text than the answer needs. Compaction keeps only the sentences that answer your query, so the main model reads a short note instead of the whole file.

Five 200-token chunks are scored and filtered to three kept sentences totalling 68 tokens, so the main model reads 68 tokens instead of 1000
Five 200-token chunks are scored and filtered to three kept sentences totalling 68 tokens, so the main model reads 68 tokens instead of 1000

Why Does This Exist?

Retrieval rarely hands you exactly what you asked for. You ask about Q3 revenue and RAG returns five chunks that mention revenue somewhere, because each chunk was embedded on its own and matched on similarity, not on whether it holds your answer. Stuff every chunk into the prompt and the main model has to wade through all of it. Every extra token adds processing, adds waiting time, and gives attention more places to look besides your answer. That's context degradation: relevant material buried under filler gets used worse.

Take a small fixture you'll check by hand. Five chunks at 200 tokens each total 1000 tokens, yet only three sentences in there answer your query. Without compaction the main model reads all 1000. With compaction it reads those three sentences at 68 tokens. Same answer, far less to read.

This page is the squeeze step between two neighbours. Context budgeting decides how many tokens each part of the prompt may use. RAG architecture shows where retrieval sits in the pipeline. Compaction is how you make what retrieval found fit the budget: score it, keep the parts that answer the query, and trim the rest before generation. You already need the basics: a context window is the fixed token frame the model can see at once, and anything outside it is simply not input.

Think of It Like This

Cutting a newspaper down to one index card

You ask a flatmate to check the business section for anything about Q3 revenue. Without compaction they drop the whole section on your table and you read every page. With compaction they read it first with scissors in hand, cut out the three paragraphs that actually answer you, and tape them to one index card. You read the card.

The cuts map onto the mechanism. Reading first is scoring. Keeping three paragraphs is thresholding. The card is the compacted prompt. Scissors never change a word, though, and that is where the analogy stops. A summarizing compactor rephrases, and rephrasing can drop a negation or round a number while the meaning looks intact. So you treat clipped sentences and rewritten summaries differently for the rest of this page.

How It Actually Works

Call the retrieved set R={c1,c2,…,cN}R = \{c_1, c_2, \ldots, c_N\}. Here NN is the chunk count, cic_i is chunk ii, and LiL_i is its length in tokens. The starting total is T=∑LiT = \sum L_i. For your query qq, each chunk gets a relevance score sis_i between 0 and 1, meaning how likely chunk ii answers qq. A threshold τ\tau (tau) is the cutoff you'll set in advance. The kept set is K={i:si≥τ}K = \{i : s_i \ge \tau\}, meaning every chunk whose score meets the cutoff. Each kept chunk is then trimmed to Li′L^\prime_i, the tokens you'll actually keep from it, where Li′≤LiL^\prime_i \le L_i. The compacted total is T′=∑i∈KLi′T^\prime = \sum_{i \in K} L^\prime_i, and the compression ratio is r=T′/Tr = T^\prime / T.

How the filter decides what stays

Scoring comes first and it is deliberately cheap. You reuse the same similarity you used for retrieval, or you ask a small model one narrow question: does this chunk answer qq? Either way you get one number per chunk, sis_i. Keep the chunk when si≥τs_i \ge \tau. A fixed cutoff adapts to easy and hard queries, since easy queries keep more and hard queries keep less, but you will see the failure this causes in a moment.

Trimming comes second and it decides Li′L^\prime_i. Extractive trimming keeps verbatim sentences that mention the query terms or the answer figure, and drops the rest of the chunk. Abstractive trimming asks a small model to restate only the query relevant facts in fewer tokens. Verbatim keeps numbers and negations intact. Rewritten summaries read shorter but can silently alter them, so prefer verbatim when the answer is a figure, a date, or a quoted condition. This two stage shape is close kin to reranking: score cheaply, spend effort only on survivors.

Worked example: five chunks become 68 tokens

Take N=5N = 5 chunks with Li=200L_i = 200 each, so T=5×200=1000T = 5 \times 200 = 1000 tokens. Suppose the scores for your query are s=[0.91,0.12,0.83,0.21,0.76]s = [0.91, 0.12, 0.83, 0.21, 0.76]. Set τ=0.5\tau = 0.5. Then K={1,3,5}K = \{1, 3, 5\}, because 0.91, 0.83, and 0.76 meet the cutoff while 0.12 and 0.21 do not. Two chunks stay, three go? No. Three stay, two go. Read that again if you skimmed: indices 1, 3, and 5 survive.

Now trim each survivor to its one answer sentence. Say chunk 1 keeps 24 tokens, chunk 3 keeps 21, and chunk 5 keeps 23. Then T′=24+21+23=68T^\prime = 24 + 21 + 23 = 68 tokens. The ratio is r=68/1000=0.068r = 68 / 1000 = 0.068, so the main model reads 6.8 percent of what retrieval returned. Savings are 1−r=0.9321 - r = 0.932, about ninety three percent fewer prompt tokens for this fixture. The diagram above draws exactly these numbers: five scored bars in, three short bars out, one small total handed onward.

What you pay for the cut

Smaller T′T^\prime buys three things: less processing per request, less waiting, and less filler competing with your answer. It charges two prices. The compactor itself takes time and can be wrong, so you have added a new component that can fail. And every dropped token is a bet that it was truly irrelevant. You lose that bet when the answer is scattered across many chunks, when qq is ambiguous and sis_i ranks the wrong chunks top, or when a rewritten summary smooths away the caveat that changed the answer.

So don't compact when the retrieved set already fits comfortably, since there's nothing to buy. Skip it or keep it verbatim when exact wording carries the answer, such as contracts, dosage instructions, or quoted policy. The assumption underneath is that relevance scores preserve answer bearing sentences. When that assumption fails, the main model answers confidently from an incomplete note, which looks exactly like a retrieval miss.

Code

A tiny filter that mirrors the worked example: score, threshold, trim, totals. No API calls, just the arithmetic the prose did by hand.

def select_chunks(scores: list[float], threshold: float) -> list[int]:    """Return indices with score at or above threshold."""    return [i for i, s in enumerate(scores) if s >= threshold]

def compacted_total(trimmed: list[int]) -> int:    return sum(trimmed)

scores: list[float] = [0.91, 0.12, 0.83, 0.21, 0.76]lengths: list[int] = [200, 200, 200, 200, 200]threshold: float = 0.5trimmed: list[int] = [24, 21, 23]  # one kept sentence per surviving chunk
kept: list[int] = select_chunks(scores, threshold)print(kept)  # -> [0, 2, 4]print(sum(lengths))  # -> 1000print(compacted_total(trimmed))  # -> 68print(round(compacted_total(trimmed) / sum(lengths), 3))  # -> 0.068

The prints agree with the hand computation: indices 0, 2, and 4 survive, 1000 tokens in, 68 out, ratio 0.068. Change τ\tau to 0.85 and only the first chunk survives, which shows how one number moves the whole result.

Watch Out For

Letting a rewrite change the answer figure

A cheap summarizer loves to smooth wording. Retrieved text says revenue excludes software sales, the summary says revenue driven by software sales, and the main model now quotes the flip with full confidence. The symptom is specific: the final answer is fluent and cites the right document, yet a number, a date, or a negation differs from the source chunk.

Keep answer bearing sentences verbatim and summarize around them. If you can't keep it verbatim, carry numbers, names, and negations across untouched, then compare the compacted note against the source before you ship the threshold change.

Setting the cutoff so high some queries keep nothing

A threshold tuned on easy queries starves hard ones. Every sis_i lands below τ\tau, KK comes back empty, and the main model answers from no context at all. It still answers, confidently, which makes this failure quiet. You will see it as correct citations on some queries and invented details on others, with nothing in between.

Always keep the top one or two chunks even when they miss the cutoff, and log how often the fallback fires. If the fallback rate climbs after you raise τ\tau, the cutoff moved, not the data. Lower it or switch to top kk for that query type.

The Quick Version

  • Context compaction shrinks retrieved text before the main model reads it, keeping only the sentences that answer the query.
  • Each chunk gets a relevance score sis_i for the query, and chunks with sis_i at or above the threshold τ\tau survive to trimming.
  • Five 200-token chunks total 1000 tokens, and three kept sentences at 24, 21, and 23 tokens leave a 68-token prompt.
  • A smaller prompt means less processing, less waiting, and less filler diluting the answer, but each compaction step can drop nuance and adds a new failure point.
  • Verbatim extracts preserve numbers and negations, while rewritten summaries read shorter and risk altering them.
  • Compaction should be skipped when retrieved text already fits, or when exact wording matters more than length.