Visual explainer
The Agent Router Architecture
Trace request classification, policy rules and confidence gating into tool, model and human lanes, then see the cost trade-off and a confident routing error.
A routing layer chooses which downstream process should handle a request. Some requests fit an approved tool workflow, some need a language model, and others need human review. Sending everything to the same model ignores those differences; routing adds a decision step with its own cost and possible mistakes.
The picture shows three representatives of a ten-request synthetic example: reset help, an essay and a refund. Their scores and costs are assumptions, not measured performance or vendor prices. This is a predefined routing workflow, even though the page calls it an agent router. Anthropic's routing pattern distinguishes that structure from an agent choosing its own sequence of actions.
Propose Before Acting
The two branches are fields of one proposal, not two executions. For the reset request, the proposed type is tool and the score is 0.96. The bar fills 96% of a zero-to-one ruler. A type names a candidate lane; the score expresses the classifier's confidence in that prediction. Neither grants permission to run a tool. The router can use rules, a small classifier or a language model; it isn't inherently cheap.
A score of 0.96 doesn't establish a 96% chance that the downstream action will succeed. Probability calibration checks whether predicted probabilities agree with observed frequencies across comparable cases. Even a calibrated prediction can be wrong on one request.
Policy Chooses the Lane
Each row traces a separate representative request: reset goes to a tool, essay to a model, and refund to a human. The lock marks a mandatory rule. Our example policy checks trusted permission and stakes information first: two flagged requests must go to a human regardless of score. For the remaining requests, it accepts the proposed tool or model lane only when the score meets the threshold; otherwise it requests human review. Tools still enforce their own authorization and input checks.
Six unflagged requests are tool candidates, with scores 0.96, 0.94, 0.92, 0.88, 0.84 and 0.60. Two are model candidates, scoring 0.93 and 0.86. The two mandatory-review requests score 0.97 and 0.89. These numbers illustrate the policy; they aren't evidence that the proposed classes are correct.
Confidence Can Defer
Set an illustrative threshold of 0.80, accepting equality. The 0.96 tool candidate passes; the 0.60 candidate goes to review. This ruler shows requests remaining after the mandatory rules. The mandatory-review request scored 0.97 still goes to a human because the earlier rule takes priority. Across all ten requests, the result is five tool dispatches, two model dispatches and three human reviews.
This threshold is a teaching choice, not a recommended default. Confidence is about the proposed classification, not the safety or correctness of a resulting answer. Routing a request to a more capable model also doesn't guarantee success.
Count the Assumed Costs
The two bars compare only model and router spending on one common scale. Assume each model call costs 75 arbitrary units and each classification costs 1 unit. Without routing, ten model calls cost 10 × 75 = 750 units. With the 0.80 policy, ten classifications plus two model calls cost 10 × 1 + 2 × 75 = 160 units. The small filled segment is the 10-unit classification cost; the rest of that bar represents 150 units of model calls. The routed subtotal is about 21.3% of the original.
Tool execution, three human reviews, retries, logging and operating costs are excluded. Those costs can outweigh the subtotal reduction, and the two workflows haven't demonstrated equal answer quality. No latency improvement follows from these bars. Evaluate the full system against a baseline that meets the same requirements; keep the simpler system when routing overhead or errors erase its benefit.
Move the Threshold
Each symbol represents one request: circles mean automatic dispatch, rounded squares mean human review. Both rows keep the same request order. The three highlighted columns change shape, showing the requests whose destinations change.
Hold the requests, scores and mandatory-review rules fixed. Raising the threshold from 0.80 to 0.90 reduces automatic dispatches from seven to four: three tool calls and one model call. Human reviews increase from three to six. The moved requests are the 0.88 and 0.84 tool candidates and the 0.86 model candidate; the classifier's predictions haven't changed. The final two columns stay under mandatory review at either threshold.
This establishes a workload change, not a quality gain. Choose a threshold using representative validation cases, the consequences of wrong routes and available review capacity. Scikit-learn's threshold-tuning guidance separates probability estimation from the decision rule and warns against tuning on the classifier's training data. Reserve separate test cases for evaluating the chosen policy.
Permission Is Separate
Both requests reach the tool lane. Only the allowed request crosses the permission boundary and runs. The denied request stops at the cross; the dashed endpoint is never reached. These are schematic process lanes, not a probability or timing plot. A high routing score cannot grant access to an account or authorize a refund. The receiving tool must validate the authenticated caller, requested operation and arguments independently.
Check Accepted Routes
This separate synthetic audit sample contains ten predictions with score 0.90: nine correct routes and one wrong route. Each square is one case, a check means correct, and a cross means wrong. Even a well-calibrated score describes a frequency across comparable predictions rather than certainty about an individual request. Ten observations alone cannot establish calibration; evaluate a larger representative sample and track changes in traffic. Audit automatic dispatches as well as the cases sent to review.
Include Review Costs
Return to the original ten requests at threshold 0.80. Add an illustrative human-review cost of 200 units per case: the routed workflow now costs 160 + 3 × 200 = 760 units before tool and operating costs, compared with the original 750-unit model subtotal. Both bars use the same scale; the three equal outlined segments are the reviews, and the mint segment is the earlier 160-unit subtotal. Break-even occurs at a review cost of (750 − 160) / 3, about 196.67 units per case. This is cost arithmetic, not proof that either workflow meets the same quality requirements.
Where It Breaks
The solid line follows a request through the gate to the crossed tool endpoint. This is a process path; horizontal position no longer measures score. The dashed path shows the required human route that wasn't chosen.
Consider one additional request outside the ten-request example. It wasn't flagged for mandatory review. The classifier incorrectly proposes the tool lane with score 0.91, so it passes either illustrated threshold. Later review establishes that the request needed a human. The error is a wrong route, not proof that a tool was authorized or that an action executed.
A confidence gate can't detect every confident mistake. Review samples of accepted routes as well as deferred ones, check calibration on current traffic, and keep independent permission checks around consequential actions. Mandatory rules only help when their inputs detect the relevant condition; confidence alone can't replace them.
The Quick Version
- One model lane ignores differences between request types.
- Classification proposes a type and confidence score before action.
- Policy rules and authorization constrain the available routes.
- Low confidence can defer requests; mandatory review takes priority.
- The example reduces a partial subtotal, not proven total costs.
- Higher thresholds can increase review workload without proving better quality.
- Tool authorization remains independent of routing confidence.
- Calibration describes groups; individual confident mistakes remain possible.
- Review costs can erase the partial subtotal saving.
- Confident routing errors can pass a confidence gate.