Context Engineering

Why Does This Exist?

Prompt engineering assumes the model already has what it needs and the only open question is how to ask. That breaks the moment a system does more than answer one self-contained question. A customer-support assistant needs a system prompt, the last dozen turns of a conversation, a couple of retrieved help-center articles, and the raw JSON an order-lookup API just returned — all read by the same model, in the same request, out of the same finite window. A sharper instruction doesn't help if the order details never made it into the prompt.

That's the gap context engineering fills. Prompt engineering writes the instruction; context engineering decides what the model actually sees when it reads that instruction — which rules, which slice of history, which passages, which tool output make it into the window, and what gets cut when they don't all fit. A context window is a budget, not a container: system instructions, history, retrieved documents, and tool output are four separate claims on the same fixed token count, and none is optional. Skip the system prompt and behavior drifts. Skip history and it re-asks for an order number it was already given. Skip retrieval and it guesses at policy. Skip tool output and it answers without ever looking at the order.

Think of It Like This

How It Actually Works

The four things competing for the window

Four categories show up in almost every production LLM system. The system prompt carries standing instructions — tone, escalation rules, what it can promise. History is everything said this session; drop it and the assistant re-asks for an order number it already has. Retrieved content — a refund-policy page, a shipping-times article — keeps answers grounded instead of guessed. Tool output is the raw result of an action just taken, and it's often the least compressible of the four, since trimming it risks cutting the field the customer actually asked about.

None of these four is a nice-to-have. Remove one and a specific failure shows up: no system prompt, drifting behavior; no history, repeated questions; no retrieval, confident wrong answers about policy; no tool output, an answer that ignores the live fact being asked about.

Budgeting: deciding what gets cut first

Add up a real system prompt, a growing conversation, a couple of retrieved articles, and a tool response, and the total reliably exceeds the window — often well before the conversation feels long. Something gives, and which of the four gives first is a design decision, not an accident.

The system prompt is usually trimmed last; it's small, and cutting it changes behavior unpredictably. Tool output is also a poor candidate for trimming, since the answer usually depends on a specific field inside it. That leaves history and retrieval to absorb the pressure, and they fail differently: trimming history means summarizing or dropping the oldest turns first; trimming a retrieved article means keeping the matching passage and cutting the rest. A well-built assistant compacts old turns into a running summary before it cuts a retrieved passage any further — a stale summary is a smaller loss than a policy article missing the clause the customer needs.

Why more context isn't free even when it fits

It's tempting to treat the problem as solved once everything technically fits under the token limit. That's the mistake. A model can have a fact fully present in context and still weight it worse than a fact near the end of the prompt, simply because thousands of tokens now sit on top of it. That's context degradation: length alone, independent of relevance, changes how well a model uses what it technically has. A detail from the system prompt can get functionally ignored once enough conversation piles on top of it — not because it left the window, but because it's no longer where attention lands.

That's why "just paste more" fails even with room to spare. Every added token competes with everything else already there, and irrelevant material dilutes attention on what actually matters. The right question is never "does this fit," it's "does adding this make the right answer more likely."

Show Me the Code

A minimal budget allocator: four requested buckets and a fixed window, trimming history first, then retrieval, until the total fits.

1from dataclasses import dataclass2
3
4@dataclass5class Budget:6    system: int7    history: int8    retrieval: int9    tool_output: int10
11
12def allocate(window: int, requested: Budget, min_system: int = 200) -> Budget:13    """Trims history first, then retrieval, to make everything fit."""14    total = requested.system + requested.history + requested.retrieval + requested.tool_output15    overflow = max(0, total - window)16    history = max(0, requested.history - overflow)              # history absorbs the cut first17    still_over = max(0, overflow - (requested.history - history))18    retrieval = max(0, requested.retrieval - still_over)         # retrieval absorbs what's left19    return Budget(max(requested.system, min_system), history, retrieval, requested.tool_output)20
21
22result = allocate(window=4000, requested=Budget(400, 2200, 1200, 500))23print(result)  # -> Budget(system=400, history=1900, retrieval=1200, tool_output=500)

Feed it the numbers from a real request and it shows exactly which bucket absorbs the overflow — history first, retrieval next, system and tool output left untouched.

Watch Out For

The Quick Version

• Prompt engineering writes the instruction; context engineering decides what the model actually gets to see when it reads it.
• Four buckets — system instructions, conversation history, retrieved documents, tool output — draw from the same fixed window, and none of them is optional.
• When the sum exceeds the window, something is cut, and the order is a design decision: history and retrieval usually absorb it before the system prompt or tool output do.
• Fitting inside the window isn't the finish line — length itself degrades how well a model attends to whatever's buried in the middle.
• "Paste more context" isn't a strategy; every added token competes with the tokens that actually matter.

What to Read Next

• Context Windows covers the hard limit context engineering budgets against.
• In-Context Learning is what the model does with whatever ends up inside the window.
• Prompt Anatomy is the instruction-writing half of the distinction this page draws.
• Memory Systems decides what history survives across sessions, feeding straight into the budgeting problem above.
• AI Agent Architecture is where this budgeting problem lives once tool calls and multi-step plans are involved.