Context Budgeting

Why Does This Exist?

Every Large Language Model (LLM) has a hard physical limit on how much text it can process in a single request, known as the Context Window.

For older models (like GPT-3), this was 4,096 tokens (~3,000 words). For modern models (like GPT-4o or Claude 3.5), it can be 128,000 or even 2 million tokens.

However, just because you can stuff 100,000 words into a prompt doesn't mean you should.

1. Cost: You are billed per input token. Sending a 100,000-token prompt costs $0.50 per request. If 1,000 users do that, you just spent$500.
2. Latency: Processing 100,000 tokens takes significantly longer than processing 1,000 tokens, degrading the user experience.
3. Performance: Sending too much irrelevant information causes Context Degradation, where the model gets confused and hallucinates.

Context Budgeting is the architectural practice of treating the context window like a strict financial budget. You explicitly assign maximum token allocations to different parts of the prompt to guarantee the system never crashes and costs remain predictable.

Think of It Like This

How It Actually Works

When building a RAG or Agentic application, the final string sent to the LLM is constructed dynamically from four main components. You must calculate the budget for each.

1. The System Prompt (Fixed Cost)

The instructions for the model (the persona, rules, and formatting). Budgeting Strategy: This is usually static. Count the tokens once during development. Let's say it's 1,000 tokens.

2. The Context/RAG Documents (Dynamic Cost)

The documents retrieved from the vector database. Budgeting Strategy: You must enforce a strict top_k limit in your vector search. If you chunk your documents into 500-token blocks, and you set top_k=5, you know the context will never exceed 2,500 tokens.

3. The Conversation History (Variable Cost)

If you are building a chatbot, you must pass the previous messages back to the LLM so it remembers what was said. If a user talks to the bot for 3 hours, this history will eventually exceed the context window. Budgeting Strategy: You must implement a sliding window (e.g., only keep the last 10 messages) or a summarizer (e.g., compress old messages into a 200-token summary).

4. The Output Room (The Hidden Cost)

If you send 127,500 tokens to a model with a 128,000 limit, the model only has 500 tokens left to generate its answer. If the answer requires 600 tokens, the LLM will abruptly cut off mid-sentence. Budgeting Strategy: You must explicitly reserve "Generation Room." You set max_tokens=2000 in the API call to guarantee the model has enough space to reply.

Show Me the Code

This code demonstrates how a backend system dynamically builds a prompt while respecting a strict token budget. Note: In production, you would use a library like tiktoken to count exact tokens; we use word counts here for simplicity (1 word $\approx$ 1.3 tokens).

1import tiktoken # Standard library for counting OpenAI tokens2
3def num_tokens_from_string(string: str, model_name: str = "gpt-4o") -> int:4    encoding = tiktoken.encoding_for_model(model_name)5    return len(encoding.encode(string))6
7def build_budgeted_prompt(system_prompt, chat_history, retrieved_docs, user_query):8    # --- STRICT BUDGET DEFINITION ---9    MAX_TOTAL_TOKENS = 800010    RESERVED_FOR_OUTPUT = 100011    AVAILABLE_PROMPT_TOKENS = MAX_TOTAL_TOKENS - RESERVED_FOR_OUTPUT12    # --------------------------------13    14    current_tokens = 015    final_prompt = ""16    17    # 1. Add System Prompt (Non-negotiable)18    sys_tokens = num_tokens_from_string(system_prompt)19    final_prompt += f"{system_prompt}\n\n"20    current_tokens += sys_tokens21    22    # 2. Add User Query (Non-negotiable)23    query_tokens = num_tokens_from_string(user_query)24    current_tokens += query_tokens25    26    # 3. Add RAG Context (Negotiable - only add what fits)27    final_prompt += "--- CONTEXT ---\n"28    for doc in retrieved_docs:29        doc_tokens = num_tokens_from_string(doc)30        if current_tokens + doc_tokens < (AVAILABLE_PROMPT_TOKENS * 0.7): # Reserve 70% of budget for context31            final_prompt += f"{doc}\n"32            current_tokens += doc_tokens33        else:34            print("Budget reached: Dropping lower-relevance RAG documents.")35            break36            37    # 4. Add Chat History (Negotiable - Slide the window)38    final_prompt += "--- CHAT HISTORY ---\n"39    # Iterate backwards to keep the most recent messages first!40    for msg in reversed(chat_history):41        msg_tokens = num_tokens_from_string(msg)42        if current_tokens + msg_tokens < AVAILABLE_PROMPT_TOKENS:43            # Prepend because we are iterating backwards44            final_prompt = final_prompt.replace("--- CHAT HISTORY ---\n", f"--- CHAT HISTORY ---\n{msg}\n")45            current_tokens += msg_tokens46        else:47            print("Budget reached: Truncating older chat history.")48            break49            50    final_prompt += f"\nUser: {user_query}\nAssistant:"51    52    print(f"\nFinal Prompt Size: {current_tokens} tokens.")53    print(f"Room left for generation: {MAX_TOTAL_TOKENS - current_tokens} tokens.")54    55    return final_prompt

Watch Out For

The Quick Version

• The Context Window is the maximum amount of text an LLM can process in one go.
• Context Budgeting is the practice of allocating strict maximums to different parts of the prompt to control costs, latency, and prevent crashes.
• The budget usually consists of: System Prompt (fixed), RAG Context (dynamic), Chat History (sliding window), and Output Room (reserved buffer).
• You must use token-counting libraries (like tiktoken) to mathematically construct the prompt before sending it to the API.

What to Read Next

• Read Context Degradation to understand why you shouldn't just send 100,000 tokens even if you can afford it.
• Read Context Compaction to see techniques for shrinking data so you can fit more information into the same budget.
• Read Episodic and Semantic Memory to see how agents manage massive chat histories without overflowing the context window.