Shadow and Canary Deployments
You'd never drive a new car off the lot before test-driving it. Shadow and canary deployments are how you test-drive a new model on real traffic — one without any users seeing the results, the other with just a small fraction — before committing to the full switch.
Why Does This Exist?
Offline evaluation is good but not good enough. Your eval set is a curated snapshot of what you expect; production traffic is a messy, unpredictable stream of what actually happens. Models that pass every offline test still fail in production — because the test set misses edge cases, because latency at scale behaves differently than in a sandbox, or because a subtle regression only appears at high concurrency.
Shadow and canary deployments let you validate a new model on real production traffic without accepting the full risk of a bad deploy. Both patterns answer the same question — "is this model safe to promote?" — but with different exposure strategies.
Think of It Like This
Think of It Like This
Imagine you've hired a new air traffic controller. You'd never just hand them the tower on their first day. First, you have them shadow an experienced controller for a week — watching all the same traffic, making the same decisions, but with the experienced controller's calls being the ones that actually go to the pilots. If the new controller looks solid after a week, you give them a quiet regional airspace, then gradually hand them more responsibility. Shadow then canary, in sequence.
Shadow Deployment
In a shadow deployment, incoming requests are duplicated and sent to both the production model (v1) and the new model (v2) simultaneously. Only v1's response is returned to the user. V2's response is logged for comparison.
This gives you:
- Zero user exposure risk — a catastrophic v2 failure (slow, wrong, OOM crash) has no user-visible impact.
- Real production request distribution — you're testing v2 against the actual traffic mix, not a curated sample.
- Side-by-side comparison — log both responses with the same request ID and compare quality scores, latency, token counts, and cost.
The cost: you're paying for two inference calls per request during the shadow period. For expensive models, shadow testing can double your inference bill temporarily. Budget for this, or cap the shadow to a sampled subset.
Canary Deployment
After shadow testing (or instead of it, for lower-risk changes), route a small slice of live traffic — typically 1–10% — to the new model. Users in the canary group get v2's real responses.
The key difference from shadow: users are now affected. This means:
- You get real feedback signal (engagement, conversion, explicit ratings if applicable).
- A bad canary creates a real user experience problem for the canary group.
- You have a clear rollback path: route 0% to v2 and 100% back to v1 in seconds.
Standard canary progression: 1% → 5% → 25% → 100%, with a wait time and metric check at each stage. Automate the promotion gates: if error rate, latency, or eval score stay within bounds at each stage, promote automatically; if any bound is breached, roll back automatically.
What to Measure During the Rollout
| Metric | Gate condition |
|---|---|
| Error rate | New model ≤ old model + 0.5% |
| P99 latency | New model ≤ old model × 1.2 |
| Eval score (sampled) | New model ≥ old model − 0.1 |
| Cost per request | New model ≤ old model × 1.5 |
Fail any gate and roll back. Pass all gates at 100% and you're done.
Watch Out For
Watch Out For
Forgetting that shadow traffic still has side effects. If your model calls external tools — sending emails, writing to a database, charging a credit card — a shadow call will execute those side effects twice. Shadow deployments work cleanly for pure-inference models. For models with tool calls or side effects, you must stub the tool calls in shadow mode or design a shadow-safe execution path. Missing this creates duplicate actions at production scale.
The Quick Version
- Shadow deployment mirrors requests to the new model but keeps all responses internal — zero user exposure.
- Canary deployment routes a small traffic percentage to the new model with real user exposure and a fast rollback path.
- Measure error rate, latency, eval score, and cost at each promotion stage; automate promotion and rollback gates.
- Shadow mode must stub side-effectful tool calls or it will duplicate actions at production scale.
What to Read Next
ab-testing-models— The formal experiment version of canary testing, with statistical significance gates and holdout control groups.incident-response-for-ml— What happens when the canary fails and you need to roll back and diagnose.