Agent Benchmarks
An agent benchmark drops a model into a sandboxed task with tools, then grades the final state, so planning and recovery count instead of just the answer.
Why Does This Exist?
A model can score 85 percent on static questions and still fail the job. Static tests ask for an answer in one shot. Real agent work asks for a sequence: open the repo, find the bug, edit the file, run the tests, read the failure, and try again. A model that writes a beautiful patch from memory but cannot read a traceback will ace the quiz and break the build.
Agent benchmarks exist to measure the doing. Each task ships with a starting state, a goal in plain words, and a checker that grades the final state. The harness resets a sandbox per task, lets the agent act for a bounded number of steps, then runs the checker. What gets scored is whether the repo ends fixed, not whether the chat sounded confident. That is why the same model can post 85 percent static and 67.5 percent interactive, and the lower number is the honest one for deployment.
This page covers how that loop is built and read. The system being scored is agent architectures. The step loop under test is ReAct. The production version of this grading, with cost and latency attached, lives in agentic workflow evaluation.
Basics first: a sandbox here is a throwaway machine holding one task, reset before the run and deleted after. The agent gets tools, a step budget, and nothing else. If sandboxes are new, the mechanism is drawn in the worked example below, and the containment reasoning sits one page over.
Think of It Like This
The driving test versus the written test
A written driving test asks what a stop sign means. Most careful readers pass. A driving test puts you in a car, in traffic, with an examiner watching the mirrors, the signals, and the recovery when you miss a turn. Plenty of people who aced the paper fail the road.
Static evaluation is the paper. The agent benchmark is the road. The sandbox is the test route, coned off so mistakes hurt nobody. The checker is the examiner with a clipboard: did you park inside the lines, not did you describe parking well. And the step limit is the fuel gauge. A driver who circles the block 40 times looking for the bay has not demonstrated skill, even if they eventually stop inside it.
Where it stops: a test route is not the city. Passing on sunny coned streets says little about ice, detours, or a screaming passenger. Benchmarks have the same limit, which is why small task counts overstate readiness.
How It Actually Works
A benchmark is three parts bolted together. The tasks are goals plus starting states, usually a few dozen to a few hundred. The environment is a sandbox per task: a repo, a browser, or a terminal with tools. The checker is a deterministic test on the final state, like a test suite passing or a file containing the right rows. Deterministic matters because it removes arguments about quality. The suite passes or it does not.
The run, step by step
The harness resets the sandbox, hands the agent the goal, and starts the clock. The agent acts, the environment answers with tool output, and the loop repeats until the agent submits or hits the step cap. Then the checker runs against the ending. A run that fixes the bug by editing the source passes. A run that writes a lovely explanation and touches nothing fails. Effort is invisible. Endings are everything.
Worked example: 27 of 40
Take a file-fix suite of 40 tasks. The agent averages 8 steps per task under a cap of 15, so the whole run costs 40 × 8 = 320 steps. It passes 27 tasks. Divide: 27 / 40 = 0.675, which is 67.5 percent. The same model scores 85 percent on a static question set, so the gap is 85 − 67.5 = 17.5 points. That gap is the price of acting: reading tool output, recovering from the 13 failures, and finishing inside the step budget. The diagram draws exactly this scoreboard.
Read it like an operator. A 67.5 percent pass rate with 8 average steps says the agent solves two thirds of jobs at a predictable cost. A 90 percent rate at 14 average steps says something else: better endings, a bill near the cap, and brittle runs that collapse the moment you tighten the budget. Always quote the rate with the step count and the cap, or the number means nothing.
This rests on two assumptions. The sandbox must actually reset, or one task leaks answers into the next. And the checker must grade outcomes, not paths, or a correct fix through an unexpected edit gets marked wrong.
Code
The point here is the harness shape: reset, loop with a cap, then grade the ending. This stdlib-only simulation scores a 40-task run with the same numbers as the worked example:
def run_benchmark(results: list[bool], steps: list[int], max_steps: int = 15) -> dict[str, float]: capped = [s if s <= max_steps else max_steps for s in steps] passed = sum(1 for ok in results if ok) rate = passed / len(results) * 100 avg_steps = sum(capped) / len(capped) return {"passed": float(passed), "rate": rate, "avg_steps": avg_steps}
results = [True] * 27 + [False] * 13steps = [8] * 40report = run_benchmark(results, steps)print(report["passed"]) # -> 27.0print(report["rate"]) # -> 67.5print(report["avg_steps"]) # -> 8.0print(85.0 - report["rate"]) # -> 17.5The outputs are the worked example restated as code: 27 passes, a 67.5 percent rate, 8 average steps, and a 17.5 point gap against the static 85. In a real harness the results list comes from the checker, not from a literal, but the division is identical.
Watch Out For
A tiny suite that rewards memorization
The symptom is a score that will not survive contact with production: 90 percent on the benchmark, confusion on the first real ticket. With only tens of tasks, the model may have seen the exact repos, versions, or pages during training, so the run measures recall, not problem solving.
Fix it by checking task provenance before trusting a number, holding out repos the model could not have seen, and treating any suite under a few hundred tasks as a smoke test. A high score on a small familiar suite is a hint, never a verdict.
A checker that grades the path instead of the ending
The symptom is a capable run marked wrong. The agent fixes the bug by setting an environment variable, but the checker only inspects one config file, so a working system fails. The score understates the agent, and tuning against it teaches the agent to repeat the expected ritual instead of fixing things.
Fix it by grading functional outcomes: run the test suite, load the page, query the data. Accept any path that leaves the state correct, and log the checker decision next to the final state so failures can be appealed.
The Quick Version
- An agent benchmark is tasks plus a reset sandbox plus a checker that grades the final state.
- Static scores flatter. A model at 85 percent static can sit at 67.5 percent interactive on 27 of 40 tasks.
- Always quote the pass rate with the average steps and the cap: 67.5 percent at 8 of 15 allowed steps.
- Deterministic checkers beat judge models because suites pass or fail without argument.
- Small familiar suites reward memorization, so verify task provenance before trusting a high score.
- Grade endings, not paths, or correct fixes through unexpected edits get marked wrong.