Agent Cost Control
Each agent step re-sends everything before it, so costs climb faster than steps. Budgets, cached text and cheaper models for easy steps keep one run affordable.
Why Does This Exist?
One chat reply bills once. An agent bills again and again, and each bill is bigger than the last.
Here is why. Providers count text in tokens, and an agent step is one model call plus the tool result that comes back (the loop in tool use and function calling). Stateless APIs do not remember, so step 2 re-sends step 1, step 3 re-sends steps 1 and 2, and so on. You pay for the same words many times.
Take the worked example below. Four steps bill 5,800 input tokens where four fresh single calls would bill 4,000. Same answers, 45% more input. Hand that loop to 1,000 users running it 10 times a day and you have billed 58 million input tokens before dinner. One stuck loop retrying all night can dwarf all of that.
This page covers the money side only: how spend adds up, and the three controls that hold it down (a hard budget, prefix reuse, cheaper models for easy steps). How the loop itself is built lives in agent harness design. What breaks, and how loops spin out, lives in agent failure modes. How you judge quality per dollar lives in agent evaluation.
You need two ideas to follow along. Tokens are the word pieces you pay for, and a step is one call plus one tool result. The links above give you both in full.
Think of It Like This
The plumber who re-reads the whole folder
You hire a plumber for a tricky leak. She keeps one folder per house: every visit note, every photo, every receipt.
She bills by the page she reads, and she re-reads the whole folder at the start of every visit. Visit one is thin. Visit four is thick, because visits one to three are inside it. You pay for the same pages again each time.
So you set three rules. You cap the job: stop at $500 no matter what. You leave the house plans on the counter so she never re-prints them. You let her apprentice handle the simple valve swap at a lower rate, and save her hours for the real diagnosis.
Mapping: the folder is the conversation history, a visit is one agent step, re-reading is re-sending history, the page rate is the token price, the $500 cap is your token budget, plans on the counter are cached prefixes, the apprentice is the cheaper model.
Where it stops: your plumber could skim or skip pages. Your model cannot. A stateless call needs the full history to stay coherent, which is why the pile-up is automatic and not laziness.
How It Actually Works
Agent cost control is the practice of predicting a run's spend before it happens and stopping or cheapening the run when it passes your limit.
The billing rule
A run with steps bills each step separately. For step , let mean input tokens sent on that step, let mean output tokens received on that step, let mean the price of one input token in dollars, and let mean the price of one output token in dollars. Total run cost , in dollars, is:
In plain words: tokens in times input price, plus tokens out times output price, added across steps. Output price sits above input price at every provider, which matters in the first pitfall below.
Why each step costs more than the last
Let mean the fixed tokens sent every step (system prompt plus task), and let mean the history tokens stored before step (past turns plus tool results). Then:
The history term is what grows. Each step appends its output and tool result to history, so climbs and every later is larger. Cost is superlinear in the step count: double the steps and you more than double the input bill.
Worked example: four steps, 5,800 tokens
Take teaching numbers, not vendor prices. Start with tokens. Each step returns output tokens and tool tokens, so history grows by per step.
- Step 1: , . History becomes .
- Step 2: , . History becomes .
- Step 3: , . History becomes .
- Step 4: , . History becomes .
Add them. Input total is tokens. Output total is tokens. Four fresh single calls with no history would bill input tokens plus the same out. The history tax is input tokens, which is extra.
To see dollars, assume purely for arithmetic an input price of $1 per million tokens and an output price of $3 per million. Then input costs dollars, output costs dollars, and dollars. The baseline without history would be $0.0064. Same arithmetic, same conclusion: history is the surcharge.
Now set a token budget of input tokens, checked before each call. After three steps the run has spent . Step 4 asks for more, and , which passes . So the harness refuses step 4 and returns what it has. That refusal is the whole technique: decide the stop line before the loop starts.
The three controls that hold spend down
- Hard budget, checked before each call. Keep a running total of tokens or converted dollars and refuse the next call when it would pass the cap. Put the check before the API call, not after, or the call that breaks you still bills. The cap also needs a per-call
max_tokenslimit, or one giant reply can jump it in a single step. - Prefix reuse. When the start of the prompt repeats exactly across steps, the provider can reuse earlier work (see prefix caching). Keep system text and documents byte-identical and first, and put the changing user text last. Reorder one line and the match fails.
- Routing easy steps to cheaper models. A small model formats JSON or extracts dates while the strong model keeps the diagnosis. Split this way inside multi-agent systems or inside one agent that picks a model per step.
Trade-off in one line: each control buys savings with a different risk. Budgets can kill a good run early. Caching helps only when prefixes truly repeat. Routing saves the most on easy steps and hurts the most when the router misjudges a hard step as easy.
Do not route when a wrong answer costs more than the tokens saved: refunds, medical or legal text, irreversible tool actions. Do not lean on caching when every prompt starts with a unique timestamp or user id. The method assumes stateless calls that re-send history; if you compact or summarize memory mid-run, shrinks and the sums above must be recomputed from the new history.
Code
The point of this snippet is a check. It replays the fixture totals and enforces the 5,000-token budget before step 4, so the prose numbers and the program agree.
def run_cost(input_tokens: list[int], output_tokens: list[int], p_in: float, p_out: float) -> float: return sum(i * p_in + o * p_out for i, o in zip(input_tokens, output_tokens))
def enforce_budget(spent_input: int, next_input: int, budget: int) -> bool: return spent_input + next_input <= budget
inputs = [1000, 1300, 1600, 1900]outputs = [200, 200, 200, 200]total = run_cost(inputs, outputs, 1e-6, 3e-6)print(f"{total:.4f}") # -> 0.0082print(sum(inputs)) # -> 5800print(enforce_budget(3900, 1900, 5000)) # -> FalseThe False is the stop signal: 3,900 spent plus 1,900 asked passes 5,000, so the caller refuses the fourth call.
Watch Out For
A token cap that ignores output prices
You cap a run at 50,000 tokens and stay under it, yet the bill jumps. Symptom: one step returns a 6,000-token JSON dump or a long report, and that single reply costs several times what its token count suggests, because output tokens bill above input tokens.
Fix with two caps, not one. Set a per-call max_tokens bound so no single reply can surprise you, and convert each usage response to dollars with your provider's current rates before the next call. Check the dollar total before sending, not after receiving.
A prefix cache that never hits
You turn on prefix caching and the hit rate sits near zero. Symptom: every step reports a cache miss even though the system prompt looks identical. The cause is almost always order or noise: a timestamp, a user id, or a reshuffled document block at the top means the prefix no longer matches from the first token, and the cache cannot apply.
Freeze the stable text first and keep it byte-identical across steps. Push timestamps, user ids and fresh tool output to the end. Log the hit rate per step and treat a flat zero as a prompt-layout bug, not bad luck.
The Quick Version
- An agent run re-sends run history on every step, so input tokens grow even when each answer stays short.
- Four steps from 1,000 starting tokens with 300 new history tokens per step bill 5,800 input tokens, not 4,000.
- A hard token budget checked before each call stops a looping run; a per-call max-tokens cap stops one giant reply.
- Prefix caching cuts bills only when the start of the prompt stays byte-identical; moving one line kills the hit.
- Routing easy steps to a cheaper model saves money when answer quality holds; hard reasoning stays on the stronger model.