Gen AI

Context Compaction

Instead of putting a massive 10,000-word document into the LLM's prompt, you use a cheaper, faster LLM to summarize and compress the document down to 500 words first, saving massive amounts of money and latency.

Why Does This Exist?

In a standard RAG application, you retrieve documents from a vector database and stuff them into the prompt.

Imagine a user asks: "What was the Q3 revenue?" The vector database retrieves a 50-page financial report because it is highly relevant. If you send that entire 50-page report (about 25,000 tokens) to GPT-4, it will cost you $0.12 and take 10 seconds to process.

However, the actual answer to the question is a single sentence on page 43: "Q3 revenue was $45M." The other 49 pages are completely irrelevant to this specific user query.

Context Compaction (also called Prompt Compression) is the process of filtering, summarizing, or condensing retrieved information before passing it to the final reasoning model. It allows you to fit much more information into your Context Budget while drastically reducing API costs.

Think of It Like This

The Executive Assistant

You are a busy CEO (the expensive, slow GPT-4 model).

Without Compaction: An employee asks you a question about a recent lawsuit. They drop a 5,000-page legal brief on your desk and say, "The answer is in there." You have to spend 8 hours reading the whole thing just to find the one relevant paragraph.

With Compaction: Your Executive Assistant (a fast, cheap model like GPT-4o-mini) reads the 5,000-page legal brief, highlights the three sentences relevant to the lawsuit, and hands you a 1-page summary. You read it in 2 minutes and answer the question perfectly.

How It Actually Works

There are three primary techniques for compacting context, ranging from simple to advanced.

1. Extractive Compaction (The Filter)

You use a fast/cheap LLM (or a traditional NLP model) to score every sentence or paragraph in the retrieved document against the user's query. If a sentence has a low relevance score, you simply delete it. The final prompt only contains the highly relevant sentences glued together.

2. Abstractive Compaction (The Summarizer)

You pass the massive document to a fast/cheap LLM and ask it to summarize the document with respect to the user's query. Prompt: "Read this financial report. The user asked about Q3 revenue. Summarize only the facts relevant to Q3 revenue, ignore everything else." The 25,000-token document becomes a 50-token summary.

3. Lexical Compaction (The Zip File)

This is an advanced, weird technique where algorithms (like LLMLingua) delete vowels, stop words (the, a, is), and redundant tokens from the text. Original: "The financial revenue for the third quarter of the year was 45 million dollars." Compacted: "financial rev Q3 45M" Because LLMs are incredibly good at pattern matching, they can still perfectly understand this "broken" English, but it costs 60% less tokens.

Show Me the Code

This code demonstrates the "Abstractive Compaction" approach using a cheap model to summarize before passing to an expensive model.

1import openai2
3def call_cheap_model(prompt):4    # e.g., GPT-4o-mini ($0.15 per 1M tokens)5    response = openai.chat.completions.create(6        model="gpt-4o-mini",7        messages=[{"role": "user", "content": prompt}]8    )9    return response.choices[0].message.content10
11def call_expensive_model(prompt):12    # e.g., GPT-4o ($5.00 per 1M tokens - 33x more expensive)13    response = openai.chat.completions.create(14        model="gpt-4o",15        messages=[{"role": "user", "content": prompt}]16    )17    return response.choices[0].message.content18
19def compacted_rag_pipeline(user_query, massive_document):20    print("1. Original Document Size: ~25,000 tokens")21    22    # --- The Compaction Step ---23    compaction_prompt = f"""24    You are an expert summarizer. Read the following massive document.25    Extract ONLY the facts that are directly relevant to answering this query: "{user_query}"26    Ignore all other information.27    28    Document: {massive_document[:1000]}... [Truncated for example]29    """30    31    compacted_context = call_cheap_model(compaction_prompt)32    print(f"2. Compacted Context Size: ~{len(compacted_context.split())} tokens")33    print(f"   Compacted Content: '{compacted_context}'\n")34    35    # --- The Final Generation Step ---36    final_prompt = f"""37    Answer the user's query based on the following context.38    39    Context: {compacted_context}40    41    Query: {user_query}42    """43    44    final_answer = call_expensive_model(final_prompt)45    print("3. Final Answer (from Expensive Model):")46    print(final_answer)47
48# --- Execution ---49user_query = "What was the Q3 revenue?"50massive_document = "Blah blah blah. [24,990 tokens of filler] Q3 revenue hit an all time high of $45M due to software sales. [More filler]"51
52compacted_rag_pipeline(user_query, massive_document)53
54# -> 1. Original Document Size: ~25,000 tokens55# -> 2. Compacted Context Size: ~12 tokens56# ->    Compacted Content: 'Q3 revenue reached $45M, driven primarily by software sales.'57# -> 58# -> 3. Final Answer (from Expensive Model):59# -> Based on the provided context, the Q3 revenue was $45 million.

Watch Out For

Loss of Nuance

Compaction is inherently a destructive process. You are throwing data away. If you use a cheap LLM to summarize a highly complex medical document, the cheap LLM might misunderstand the medical jargon and omit a critical nuance from its summary. When the expensive model reads the flawed summary, it will give the wrong answer. Always evaluate if your domain requires full-text accuracy or if it can survive summarization.

The Quick Version

Large context windows allow you to pass massive documents to LLMs, but doing so is incredibly expensive and slow.
Most of the text in a retrieved document is irrelevant to the specific user query.
Context Compaction uses cheap models (or algorithms) to filter, summarize, or shrink the document before sending it to the expensive reasoning model.
It acts like an Executive Assistant, saving the CEO (the main LLM) from having to read 50 pages of irrelevant filler.