Skip to content
AI360XpertBeta

Agent Cost Control

Each agent step re-sends everything before it, so costs climb faster than steps. Budgets, cached text and cheaper models for easy steps keep one run affordable.

Each agent step re-sends its growing history, so input tokens pile up until a fixed budget stops the run.
Each agent step re-sends its growing history, so input tokens pile up until a fixed budget stops the run.

Why Does This Exist?

One chat reply bills once. An agent bills again and again, and each bill is bigger than the last.

Here is why. Providers count text in tokens, and an agent step is one model call plus the tool result that comes back (the loop in tool use and function calling). Stateless APIs do not remember, so step 2 re-sends step 1, step 3 re-sends steps 1 and 2, and so on. You pay for the same words many times.

Take the worked example below. Four steps bill 5,800 input tokens where four fresh single calls would bill 4,000. Same answers, 45% more input. Hand that loop to 1,000 users running it 10 times a day and you have billed 58 million input tokens before dinner. One stuck loop retrying all night can dwarf all of that.

This page covers the money side only: how spend adds up, and the three controls that hold it down (a hard budget, prefix reuse, cheaper models for easy steps). How the loop itself is built lives in agent harness design. What breaks, and how loops spin out, lives in agent failure modes. How you judge quality per dollar lives in agent evaluation.

You need two ideas to follow along. Tokens are the word pieces you pay for, and a step is one call plus one tool result. The links above give you both in full.

Think of It Like This

The plumber who re-reads the whole folder

You hire a plumber for a tricky leak. She keeps one folder per house: every visit note, every photo, every receipt.

She bills by the page she reads, and she re-reads the whole folder at the start of every visit. Visit one is thin. Visit four is thick, because visits one to three are inside it. You pay for the same pages again each time.

So you set three rules. You cap the job: stop at $500 no matter what. You leave the house plans on the counter so she never re-prints them. You let her apprentice handle the simple valve swap at a lower rate, and save her hours for the real diagnosis.

Mapping: the folder is the conversation history, a visit is one agent step, re-reading is re-sending history, the page rate is the token price, the $500 cap is your token budget, plans on the counter are cached prefixes, the apprentice is the cheaper model.

Where it stops: your plumber could skim or skip pages. Your model cannot. A stateless call needs the full history to stay coherent, which is why the pile-up is automatic and not laziness.

How It Actually Works

Agent cost control is the practice of predicting a run's spend before it happens and stopping or cheapening the run when it passes your limit.

The billing rule

A run with nn steps bills each step separately. For step ii, let IiI_i mean input tokens sent on that step, let OiO_i mean output tokens received on that step, let pinp_{in} mean the price of one input token in dollars, and let poutp_{out} mean the price of one output token in dollars. Total run cost CC, in dollars, is:

C=∑i=1n(Ii⋅pin+Oi⋅pout)C = \sum_{i=1}^{n} (I_i \cdot p_{in} + O_i \cdot p_{out})

In plain words: tokens in times input price, plus tokens out times output price, added across steps. Output price sits above input price at every provider, which matters in the first pitfall below.

Why each step costs more than the last

Let SS mean the fixed tokens sent every step (system prompt plus task), and let Hi−1H_{i-1} mean the history tokens stored before step ii (past turns plus tool results). Then:

Ii=S+Hi−1I_i = S + H_{i-1}

The history term is what grows. Each step appends its output and tool result to history, so HH climbs and every later IiI_i is larger. Cost is superlinear in the step count: double the steps and you more than double the input bill.

Worked example: four steps, 5,800 tokens

Take teaching numbers, not vendor prices. Start with S=1000S = 1000 tokens. Each step returns 200200 output tokens and 100100 tool tokens, so history grows by 300300 per step.

  • Step 1: I1=1000I_1 = 1000, O1=200O_1 = 200. History becomes 300300.
  • Step 2: I2=1000+300=1300I_2 = 1000 + 300 = 1300, O2=200O_2 = 200. History becomes 600600.
  • Step 3: I3=1000+600=1600I_3 = 1000 + 600 = 1600, O3=200O_3 = 200. History becomes 900900.
  • Step 4: I4=1000+900=1900I_4 = 1000 + 900 = 1900, O4=200O_4 = 200. History becomes 12001200.

Add them. Input total is 1000+1300+1600+1900=58001000 + 1300 + 1600 + 1900 = 5800 tokens. Output total is 4×200=8004 \times 200 = 800 tokens. Four fresh single calls with no history would bill 4×1000=40004 \times 1000 = 4000 input tokens plus the same 800800 out. The history tax is 5800−4000=18005800 - 4000 = 1800 input tokens, which is 1800/4000=45%1800 / 4000 = 45\% extra.

To see dollars, assume purely for arithmetic an input price of $1 per million tokens and an output price of $3 per million. Then input costs 5800×10−6=0.00585800 \times 10^{-6} = 0.0058 dollars, output costs 800×3×10−6=0.0024800 \times 3 \times 10^{-6} = 0.0024 dollars, and C=0.0082C = 0.0082 dollars. The baseline without history would be $0.0064. Same arithmetic, same conclusion: history is the surcharge.

Now set a token budget of B=5000B = 5000 input tokens, checked before each call. After three steps the run has spent 1000+1300+1600=39001000 + 1300 + 1600 = 3900. Step 4 asks for 19001900 more, and 3900+1900=58003900 + 1900 = 5800, which passes 50005000. So the harness refuses step 4 and returns what it has. That refusal is the whole technique: decide the stop line before the loop starts.

The three controls that hold spend down

  • Hard budget, checked before each call. Keep a running total of tokens or converted dollars and refuse the next call when it would pass the cap. Put the check before the API call, not after, or the call that breaks you still bills. The cap also needs a per-call max_tokens limit, or one giant reply can jump it in a single step.
  • Prefix reuse. When the start of the prompt repeats exactly across steps, the provider can reuse earlier work (see prefix caching). Keep system text and documents byte-identical and first, and put the changing user text last. Reorder one line and the match fails.
  • Routing easy steps to cheaper models. A small model formats JSON or extracts dates while the strong model keeps the diagnosis. Split this way inside multi-agent systems or inside one agent that picks a model per step.

Trade-off in one line: each control buys savings with a different risk. Budgets can kill a good run early. Caching helps only when prefixes truly repeat. Routing saves the most on easy steps and hurts the most when the router misjudges a hard step as easy.

Do not route when a wrong answer costs more than the tokens saved: refunds, medical or legal text, irreversible tool actions. Do not lean on caching when every prompt starts with a unique timestamp or user id. The method assumes stateless calls that re-send history; if you compact or summarize memory mid-run, HH shrinks and the sums above must be recomputed from the new history.

Code

The point of this snippet is a check. It replays the fixture totals and enforces the 5,000-token budget before step 4, so the prose numbers and the program agree.

def run_cost(input_tokens: list[int], output_tokens: list[int], p_in: float, p_out: float) -> float:    return sum(i * p_in + o * p_out for i, o in zip(input_tokens, output_tokens))

def enforce_budget(spent_input: int, next_input: int, budget: int) -> bool:    return spent_input + next_input <= budget

inputs = [1000, 1300, 1600, 1900]outputs = [200, 200, 200, 200]total = run_cost(inputs, outputs, 1e-6, 3e-6)print(f"{total:.4f}")  # -> 0.0082print(sum(inputs))  # -> 5800print(enforce_budget(3900, 1900, 5000))  # -> False

The False is the stop signal: 3,900 spent plus 1,900 asked passes 5,000, so the caller refuses the fourth call.

Watch Out For

A token cap that ignores output prices

You cap a run at 50,000 tokens and stay under it, yet the bill jumps. Symptom: one step returns a 6,000-token JSON dump or a long report, and that single reply costs several times what its token count suggests, because output tokens bill above input tokens.

Fix with two caps, not one. Set a per-call max_tokens bound so no single reply can surprise you, and convert each usage response to dollars with your provider's current rates before the next call. Check the dollar total before sending, not after receiving.

A prefix cache that never hits

You turn on prefix caching and the hit rate sits near zero. Symptom: every step reports a cache miss even though the system prompt looks identical. The cause is almost always order or noise: a timestamp, a user id, or a reshuffled document block at the top means the prefix no longer matches from the first token, and the cache cannot apply.

Freeze the stable text first and keep it byte-identical across steps. Push timestamps, user ids and fresh tool output to the end. Log the hit rate per step and treat a flat zero as a prompt-layout bug, not bad luck.

The Quick Version

  • An agent run re-sends run history on every step, so input tokens grow even when each answer stays short.
  • Four steps from 1,000 starting tokens with 300 new history tokens per step bill 5,800 input tokens, not 4,000.
  • A hard token budget checked before each call stops a looping run; a per-call max-tokens cap stops one giant reply.
  • Prefix caching cuts bills only when the start of the prompt stays byte-identical; moving one line kills the hit.
  • Routing easy steps to a cheaper model saves money when answer quality holds; hard reasoning stays on the stronger model.