Core ML

Model Routing

Not every question deserves a $20 answer. Model routing looks at each incoming request, decides how hard it is, and sends it to the cheapest model that can handle it — saving 40-60% on inference costs in real applications without users noticing.

A model router classifies each incoming request by complexity and routes simple questions to a cheap fast model while sending hard reasoning tasks to a large expensive model, cutting costs without changing accuracy.

Why Does This Exist?

Using GPT-4o for every request is like driving a Formula 1 car to the grocery store. It works, but it costs roughly 33× more than a family sedan for the same outcome. In a real LLM application, the majority of requests — status questions, simple lookups, short summaries, format conversions — don't need the largest most capable model. They need a correct answer quickly.

Model routing is the component that makes this distinction automatically. It sits in front of your models and decides, per request, which model should handle it.

Think of It Like This

A law firm doesn't have a senior partner answer every email. The receptionist handles "what are your office hours?", a paralegal handles document requests, and the partner handles courtroom strategy. Each question gets routed to the cheapest person capable of answering it correctly. The client doesn't care — they just get the right answer.

How It Actually Works

There are three main routing strategies, each with a different cost/accuracy profile.

1. Rule-based routing

The simplest version. Hard-code routing decisions based on measurable signal:

Token count: if the prompt is under 200 tokens, use the small model.
Task type: if the system prompt says "translate" or "summarise", use the small model. If it says "reason about" or "critique", use the large one.
User tier: paying users get the large model; free users get the small one.

This is fast (no classifier overhead), easy to debug, and surprisingly effective. It breaks when your task distribution shifts or when "long prompt" no longer correlates with "hard task."

2. Classifier-based routing

Train or prompt a small classifier to predict task difficulty before routing. The classifier reads the incoming message and outputs a routing decision. RouteLLM, the open-source router from LMSYS, trains on human preference data and achieves 40–50% cost reduction with less than 1% accuracy loss on MMLU.

The classifier runs on CPU in 2–5ms, so it adds negligible latency. The tricky part is keeping the classifier calibrated as your user base and task distribution change over time.

3. Cascade (progressive retry)

Send every request to the small model first. If the confidence score is below a threshold, fall through to the large model. This works without any separate classifier — the small model itself signals when it's uncertain.

The downside is latency: you always pay the small-model inference time before deciding to escalate. For latency-sensitive apps, this is a deal-breaker. For async pipelines, it's often the simplest approach.

Show Me the Code

A simple rule-and-classifier hybrid router:

1from enum import Enum2
3class Model(Enum):4    SMALL = "gpt-4o-mini-2024-07-18"   # $0.15/1M input tokens5    LARGE = "gpt-4o-2024-11-20"        # $2.50/1M input tokens6
7SIMPLE_KEYWORDS = {"translate", "summarise", "summarize", "list", "what is", "define"}8
9def route(prompt: str, system_prompt: str = "") -> Model:10    # Rule 1: very short prompts are almost always simple11    if len(prompt.split()) < 30:12        return Model.SMALL13
14    # Rule 2: keyword signal from the task type15    lower = (prompt + " " + system_prompt).lower()16    if any(kw in lower for kw in SIMPLE_KEYWORDS):17        return Model.SMALL18
19    # Rule 3: reasoning / code tasks need the large model20    hard_signals = {"reason", "explain why", "debug", "refactor", "compare", "critique"}21    if any(sig in lower for sig in hard_signals):22        return Model.LARGE23
24    # Default: small model, cascade to large if confidence is low25    return Model.SMALL26
27
28def call_with_routing(prompt: str) -> str:29    model = route(prompt)30    response = call_llm(prompt, model=model.value)31
32    # Cascade: if small model output looks uncertain, retry with large33    if model == Model.SMALL and is_uncertain(response):34        response = call_llm(prompt, model=Model.LARGE.value)35
36    return response

Watch Out For

Accuracy regression in the wrong slice. Aggregate accuracy metrics will hide routing failures. Your overall accuracy might be 92% whether you route or not, but the 8% you're getting wrong could be systematically in the subset now being routed to the small model. Before deploying a router, slice your eval set by task type and measure small-model accuracy on each slice separately. A router that drops accuracy on your highest-value use case is worse than no router at all.

The Quick Version

Model routing sends each request to the cheapest model that can answer it correctly.
Three strategies: rule-based (fast, fragile), classifier-based (accurate, needs calibration), cascade (simple, slower).
In typical production traffic, 60–70% of requests can go to the small model without accuracy loss.
Always evaluate routing accuracy by task slice, not just overall — a smart router creates new failure modes.