Skip to content
AI360XpertBeta

Context Budgeting

Each request gets a fixed number of tokens to split. Budgeting decides how many go to instructions, retrieved passages, chat history, and the answer itself.

A fixed 8,000-token window splits into metered lanes: 800 for system instructions, 120 for the query, 1,500 for history, 4,500 for five retrieved chunks, and 1,000 reserved for the answer, leaving 80 spare.
A fixed 8,000-token window splits into metered lanes: 800 for system instructions, 120 for the query, 1,500 for history, 4,500 for five retrieved chunks, and 1,000 reserved for the answer, leaving 80 spare.

Why Does This Exist?

Every model call sees through a context window: a fixed frame of tokens that must hold the entire input plus the answer. That prompt is never one blob you wrote by hand. Static instructions, the current question, recent chat history, and freshly retrieved passages all pour into the same frame, and the dynamic pieces change size on every request.

Two things break when you just concatenate and hope. The loud one is the hard wall. The request no longer fits, so the call fails or the tail gets silently cut. The quiet one costs more. An oversized prompt burns tokens you pay for, answers slower, and buries the one passage that matters under filler. Stuffing everything in feels safe. It performs worse.

A context budget is the fix: a token allowance per lane, enforced before the call goes out. This page covers the split itself. How to carve the frame into lanes with caps, in what order to fill them, and what to drop first. It assumes you know what the frame is and how text becomes countable units. If either is fuzzy, read context windows and tokenization first. What happens when a full prompt still answers badly is context degradation. How to shrink pieces that will not fit is context compaction.

Think of It Like This

Packing a suitcase with a strict weight limit

You fly with a 20-kilo allowance and four piles: documents you must carry, clothes for the days ahead, books you might read, and empty space you need for the return trip. You weigh the mandatory pile first, then add clothes newest on top, then books in order of want, stopping the moment the scale reads full. The books left on the bed are not lost. They just did not make this flight.

The mapping holds point for point. The allowance is the window. The mandatory pile is the system prompt. Clothes are chat history, newest first. Books are retrieved passages in rank order. Empty space is the reserved answer room, weighed before the books, never after.

It stops where luggage stops. Clothes do not change meaning when folded in a different order, and leaving a shirt behind never corrupts the shirts you packed. Prompt pieces interact. Order, duplication, and one missing sentence can change what the model concludes, so dropping the sixth chunk is safe only when the kept five still contain the answer.

How It Actually Works

The budget rule

Name the pieces. WW is the window: the most tokens one request may hold, input plus output. SS is the system prompt, the fixed instructions the model always sees. QQ is the current user message. HH is the history you keep, newest messages first. D1,D2,…D_1, D_2, \ldots are retrieved chunks in rank order, best first. RR is the room you reserve for the answer before filling anything else.

The rule is one inequality:

S+Q+H+∑Dkept+R≤WS + Q + H + \sum D_{\text{kept}} + R \le W

In plain words: fixed pieces plus kept history plus kept chunks plus reserved answer room must fit the window. Two habits make it hold. First, subtract RR before you start filling, because output you never reserved is output that gets cut mid-sentence. Second, fill negotiable lanes in priority order and stop at the cap: newest history first, then chunks best first, dropping whatever does not fit.

Filling the lanes

Fixed lanes go in untouched. SS and QQ are non-negotiable, so measure them and move on. History gets a cap, not a promise: keep the newest turns up to your allowance and summarize or drop the rest. Docs get whatever the inequality still allows: walk the ranking, keep a chunk when it fits, and stop at the first one that does not. That first failure ends the walk. Lower-ranked chunks behind it stay out too, because skipping past a long chunk to grab a short one trades relevance for packing efficiency, and relevance is the point.

The trade is explicit. Every token given to history or retrieval is a token the answer cannot use, and long prompts cost more and respond slower. When the prompt is a frozen template with no retrieval and no memory, say a one-shot classifier, lane machinery is overkill: measure the template once and keep a fixed margin. Budgeting earns its keep the moment any lane varies per request, which is exactly what RAG architecture does on every call.

The scheme rests on one assumption: you can count tokens before sending, and every lane holds pieces you may drop or shrink. It fails on a single unsplittable blob, like one giant tool dump poured into history. No cap can fix that shape. Split the blob into chunks or compress it first, then budget the pieces.

Worked example

Take a window of W=8,000W = 8{,}000 tokens. Your fixed costs are S=800S = 800 for instructions and Q=120Q = 120 for the question. You keep H=1,500H = 1{,}500 tokens of recent history. You reserve R=1,000R = 1{,}000 for the answer.

Room for retrieved chunks is what remains:

8,000−1,000−800−120−1,500=4,5808{,}000 - 1{,}000 - 800 - 120 - 1{,}500 = 4{,}580

Retrieval returned six 900-token chunks in rank order. Walk them: the first five total 5×900=4,5005 \times 900 = 4{,}500, which fits inside 4,580. The sixth needs 900 and only 80 remain, so it stays out along with anything behind it.

Check the whole request. Input tokens are 800+120+1,500+4,500=6,920800 + 120 + 1{,}500 + 4{,}500 = 6{,}920. Add the reserved 1,000 and you get 7,920 against the 8,000 window, 80 to spare. That is the diagram above: five kept chunks, one dropped, and a thin sliver of headroom proving the fit is tight but real.

Code

The point of this snippet is the reserve-first habit: subtract the answer room before filling anything, then walk ranked chunks and stop at the first one that misses.

def fit_docs(    window: int, reserved: int, fixed: list[int], ranked_docs: list[int]) -> tuple[list[int], int, int]:    """Keep top-ranked chunks that fit; stop at the first miss."""    capacity = window - reserved - sum(fixed)    kept: list[int] = []    used = 0    for size in ranked_docs:        if used + size > capacity:            break        kept.append(size)        used += size    return kept, used, capacity - used

window, reserved = 8000, 1000fixed = [800, 120, 1500]  # system, query, historyranked = [900, 900, 900, 900, 900, 900]
kept, docs_used, docs_spare = fit_docs(window, reserved, fixed, ranked)total_input = sum(fixed) + docs_usedprint(kept)                # -> [900, 900, 900, 900, 900]print(docs_spare)          # -> 80print(total_input)         # -> 6920print(total_input + reserved <= window)  # -> True

The sixth chunk never enters the list, which is the behavior you want: rank order decides, not packing luck.

Watch Out For

Reserving output room last

Your prompts fit, your tests pass, and production answers end mid-sentence. The tell is the stop reason: the call reports length instead of a clean finish, always on the longest answers and never on short ones.

The cause is order of operations. Filling every lane to the brim and hoping some room remains is not reserving. On a long question the hope fails exactly when the answer matters most. Subtract RR from the window before any lane fills, as the snippet does, and treat a length stop as a budgeting alarm, not a model flaw.

Counting words instead of tokens

Local math says the prompt fits and production disagrees. The symptom is intermittent: short plain-English prompts pass while code, JSON tool output, or multilingual text overflows the same budget.

Words and characters are not tokens. Code and structured output tokenize denser than prose, so a 900-word JSON dump can cost far more than 900 tokens. Count with the tokenizer of the model you call, recount after final assembly, and cap each lane in tokens, never in messages or characters.

The Quick Version

  • A context budget is a token allowance per lane: system instructions, the query, history, retrieved chunks, plus answer room subtracted from the window before negotiable lanes fill.
  • The rule is S+Q+H+kept docs+R≤WS + Q + H + \text{kept docs} + R \le W. Fill fixed lanes first, then newest history, then best-ranked chunks, stopping at the first miss.
  • Count with the real tokenizer. Words, characters, and message counts all lie about tokens.
  • A full window is not the goal. Each extra chunk adds cost and latency and can bury the passage that holds the answer.
  • When one blob will not fit its lane, split it or compress it first. No cap fixes an unsplittable lump.