Agent Harness Design
The wrapper around a looping agent that sets the stops the model never sets itself: how many steps, how much time, which errors get a retry, and what it all cost.
Why Does This Exist?
A bare agent loop has no sense of enough. It thinks, calls a tool, reads the result, and goes again. Nothing in that cycle counts steps, watches the clock, or asks what the run has cost so far. When a tool keeps failing, or the model keeps rephrasing the same bad call, the loop just keeps spending. You don't hear about it from the loop. You hear about it from the bill, or from a user who waited a minute for nothing.
The fix is ordinary code around the loop: the agent harness. It counts every step, enforces a wall-clock budget, sorts each tool error into retry or stop, and writes every token to a running ledger. Think of the rule it enforces as "run until the model says stop, or step 8, or 90 seconds, whichever comes first." Short. Testable. No judgment calls at 2 a.m.
Keep two neighbours straight. Agent sandboxing decides what the agent may touch. Agent evaluation scores whether past runs went well. This page decides when the current run ends and what happens the moment a tool call blows up.
One idea first, briefly. The loop being wrapped is the ReAct pattern: think, act through a tool, observe the result, repeat. If that cycle is new, read that page first and come back. Nothing here picks actions. The harness only bounds the loop that does.
Think of It Like This
A climber with a belayer
You are belaying a friend on an indoor wall. She climbs. You hold the rope and you keep the log.
The rope length is the step cap. She gets that many moves, then she comes down whether she reached the top or not. The clock on the wall is the timeout. When the session ends, it ends mid-move if it must. When she slips on a move, the rope catches her and she tries the move again from just below. That is a retryable error. When the hold itself spins loose in her hand, you call the climb off. Nobody retries a broken hold. That is a fatal error. And your pencil log of attempts and minutes is the ledger. Nobody remembers the session exactly, so you wrote it down as it happened.
Where it stops: you cannot teach her footwork from the ground. A belayer limits falls and keeps time. The climbing is still hers.
How It Actually Works
Four jobs, all in plain code around the loop: count steps, watch time, sort errors, and write the ledger. The model still chooses every action. The harness only decides whether the run continues.
The loop under control
Name the state once and the rest follows. Let be steps used so far, the step cap, elapsed wall-clock seconds, the time budget, tokens spent so far, and the token budget. In plain words: three counters with three ceilings.
Before each model call the gate checks two inequalities: and . After each step the ledger adds that step's input plus output tokens into and checks . Any check that fails ends the run at once with a fixed message, not with another model call. No negotiation. That ordering matters: the gate runs before the model, because a model that is already stuck cannot be trusted to stop itself.
The worked run
Fix small numbers and follow them. Set steps and seconds.
- Step 1: the agent looks up a customer id. Tokens: 1,200. Time: 12 s. Tool says ok. Now , , .
- Step 2: the agent passes a malformed date to the same tool. Tokens: 950. Time: 9 s. Tool returns an error string. The harness classes it retryable and hands the text back as the next observation. Now , , .
- Step 3: the agent fixes the date and retries. Tokens: 1,400. Time: 11 s. Tool says ok. Now , , .
Same arithmetic the diagram draws: 3,550 tokens spent, 32 seconds gone, 58 seconds left (), 5 steps left (). Had the date stayed broken, the loop would have burned its remaining 5 steps on the same mistake and stopped at by rule. That is the whole point: the worst case is known before the run starts.
Error sorting
Not every failure deserves a retry. A retryable error means the call was well formed but wrong: bad date string, misspelled id, empty result the agent can work around. The harness converts it to observation text and lets the loop continue. A fatal error means the system underneath is broken: connection refused, permission denied, credentials expired. Retrying those teaches nothing and spends tokens on a dead end, so the harness ends the run at once and says so. If you take one habit from this page, take this split. Most runaway bills come from retrying fatal errors with a hopeful model, as agent failure modes shows in detail.
The ledger and the trade-off
The ledger is just addition, done honestly: every step adds its input tokens plus its output tokens into , and the total ships with the final answer. That number is what you alert on, chart, and charge against. It assumes tool failures arrive as values the code can read. A tool that hangs past its own timeout, or writes a side effect before failing, breaks that assumption. Counting cannot un-send an email, which is why timeouts and agent sandboxing sit next to the ledger rather than under it.
Tighter caps cut cost but kill long honest tasks; looser caps finish those tasks and let confused runs spend more. Pick caps from your own traces: the step count and token total your good runs actually need, plus a small margin. And skip the whole apparatus for a single model call with no tools. No loop, no harness. Plain validation does that job better.
Code
The point of this snippet is the gate plus the ledger as runnable state, not pseudocode. Three steps go in with the fixture's numbers, and the totals come out matching the worked run.
from dataclasses import dataclass
@dataclassclass HarnessLedger: max_steps: int max_seconds: float steps_used: int = 0 seconds_used: float = 0.0 tokens_used: int = 0
def note_step(self, tokens: int, seconds: float) -> str: self.steps_used += 1 self.tokens_used += tokens self.seconds_used += seconds if self.steps_used >= self.max_steps: return "stop: step cap" if self.seconds_used >= self.max_seconds: return "stop: time budget" return "go"
ledger = HarnessLedger(max_steps=8, max_seconds=90.0)print(ledger.note_step(1200, 12.0)) # -> goprint(ledger.note_step(950, 9.0)) # -> goprint(ledger.note_step(1400, 11.0)) # -> goprint((ledger.tokens_used, ledger.seconds_used, ledger.steps_used)) # -> (3550, 32.0, 3)Three calls return go because 3 steps sit under the cap of 8 and 32 seconds sit under 90. The tuple reads (3550, 32.0, 3): the same 3,550 tokens and 32 seconds the worked run computed by hand. Push note_step five more times and the eighth call returns stop: step cap, which is the run ending by rule.
Watch Out For
Retrying a dead system until the cap
Symptom: every step returns the same connection or permission error, the agent rephrases and retries, and the run burns all 8 steps before stopping. The ledger fills with tokens that never had a chance.
Fix: sort before you retry. Match fatal phrases (refused, denied, unauthorized, timed out at the socket) to an instant stop, and only feed back errors the agent can act on, like a bad argument or an empty result. Log which branch each error took so the next trace shows the split working.
A step cap with no token or time cap
Symptom: the run stays under 8 steps and still hurts. One step pulls a huge page into context or waits on a slow tool, and tokens or latency blow past anything the step count suggested.
Fix: enforce all three ceilings together, exactly as the gate above does. Give every tool call its own short timeout as well, so one slow call cannot eat the whole 90 seconds. A cap you don't enforce in code is a wish.
The Quick Version
- An agent harness is the wrapper around a think-act-observe loop that enforces the stops the model never enforces on itself.
- Step caps, timeouts, and token budgets trip together in the gate; any single cap alone still leaves a hole.
- Retryable tool errors return to the model as plain observation text; broken systems end the run at once.
- A token ledger adds each step's input and output tokens into one running total, such as 3,550 after the three fixture steps.
- Error sorting decides each failure's branch: fixable input goes back for another try, dead systems stop the clock.