Agentic Workflow Evaluation

Why Does This Exist?

You asked for a CSV of three competitor prices. One agent returns it after 3 tool calls. Another returns the same CSV after 11 calls, two of them errors, and a lucky guess on the last price. If you grade only the file, both pass. The second one still burns ten times the tokens and will fail the next time luck runs out.

That is why final-answer checks are not enough for agents. An agent makes a sequence of choices: what to try, which tool to call, what the tool said, and whether to retry. Chain-of-thought agents write that reasoning down, code execution agents add sandbox outputs to it, and human-in-the-loop agents pause it for approval. Evaluation has to read that whole trace.

This page covers scoring the trace: outcome, efficiency, and errors. It does not teach how to write the reasoning trace itself or when to ask a person to step in. Those live on their own pages, linked above.

Basics first: a tool call is one request from the agent to outside code, plus the reply that comes back. If that idea is new, skim tool use first, then come back. Everything below assumes you know that a trace is just the ordered list of those calls.

Think of It Like This

How It Actually Works

Call the step count $n$, which means how many tool calls the trace holds. Call the outcome score $o$, which means how good the final answer is on a 0 to 1 scale. Call the error count $e$, which means how many steps returned an error. The holistic score combines all three, because any one of them lies on its own.

Outcome first, then the path

Outcome asks whether the goal changed the world as requested. For a CSV that means the right rows and columns. For a subjective summary it often means another model grades against a short rubric. Outcome alone passes the lucky 11-step run, which is why you never stop here.

Efficiency and error penalties

Efficiency falls as steps grow past what the task should need. Fix an expectation, here 5 steps. Then efficiency is $1.0 - (n - 5) \times 0.15$, clipped to the range 0 to 1. Three steps gives $1.0 - (3 - 5) \times 0.15 = 1.30$, clipped to 1.0. Eleven steps gives $1.0 - (11 - 5) \times 0.15 = 0.10$. Each error subtracts another 0.2, because retries cost money and hint at confused reasoning.

The final score is $0.6 \times o + 0.4 \times \text{efficiency} - 0.2 \times e$, floored at 0. Outcome carries more weight, but a messy path can still drag a good answer down.

Worked example

Task: write three prices into a CSV.

Tidy run: $n = 3$, $e = 0$, $o = 1.0$. Efficiency is 1.0. Final is $0.6 \times 1.0 + 0.4 \times 1.0 - 0 = 1.0$.

Wandering run: $n = 11$, $e = 2$, $o = 0.9$. Efficiency is 0.1. Final is $0.6 \times 0.9 + 0.4 \times 0.1 - 0.2 \times 2 = 0.54 + 0.04 - 0.40 = 0.18$.

Same file, very different grades. The 0.82 gap is the price of eight extra steps and two errors. That gap is what lets you compare prompts, models, and retry policies without fooling yourself.

Code

This scorer implements the rule above with only the standard library. It takes counts, not live models, so you can check the arithmetic by hand:

1def efficiency(n_steps: int, expected: int = 5) -> float:2    raw = 1.0 - (n_steps - expected) * 0.153    return max(0.0, min(1.0, raw))4
5
6def holistic(outcome: float, n_steps: int, n_errors: int) -> float:7    eff = efficiency(n_steps)8    return max(0.0, outcome * 0.6 + eff * 0.4 - n_errors * 0.2)9
10
11print(round(efficiency(3), 2))  # -> 1.012print(round(efficiency(11), 2))  # -> 0.113print(round(holistic(1.0, 3, 0), 2))  # -> 1.014print(round(holistic(0.9, 11, 2), 2))  # -> 0.18

Four calls, two stories. The tidy trace keeps its 1.0. The wandering trace keeps its 0.9 outcome but finishes at 0.18 once steps and errors are counted.

Watch Out For

The Quick Version

• Agentic workflow evaluation scores the steps plus the final answer, because the same file can come from a tidy or a lucky path.
• Outcome $o$ says whether the goal was met, step count $n$ says what it cost, error count $e$ says how confused the run was.
• Efficiency is $1.0 - (n - 5) \times 0.15$ clipped to 0 to 1, and each error subtracts 0.2 more.
• A 3-step clean run with outcome 1.0 scores 1.0, while an 11-step run with two errors and outcome 0.9 scores 0.18.
• Do not punish a single smart retry the same way you punish a five-time repeat.
• Model judges prefer long traces, so pair them with a rubric and human spot checks.

What to Read Next

• Chain of Thought Agents
• Code Execution Agents
• Human-in-the-Loop Agents