Canary Deployments in ML

A new model serves a small slice of real traffic first. If it misbehaves, only that slice feels it, and the rest stay on the proven model.

Ninety percent of traffic stays on baseline v1 while ten percent probes canary v2, and the error comparison decides whether rollout continues.

Why Does This Exist?

Models fail in ways unit tests cannot see. A new version can load fine, answer every schema check, and still burn twice the memory, answer half as fast, or go quietly wrong on one slice of inputs. Offline validation never felt production load or production data, so it never had the chance to catch any of that.

Shipping to everyone at once turns each of those misses into a full outage. The canary limits the blast radius: v2 serves a small fraction, say 5 or 10 percent, while v1 carries the rest. Bad behavior then shows up in the canary slice first, where reverting costs one routing change. This page covers the gradual rollout. The one-flip alternative is blue-green deployments, and long-term decay after release is model drift detection.

Basics first: sticky routing sends each user to the same model for the whole experiment, usually by hashing the user id. Without it, one person bounces between v1 and v2 across requests, which ruins recommendations and makes the metrics unreadable. Every canary below assumes stickiness.

Think of It Like This

The new soup goes to five tables first

A restaurant trusts a new soup recipe but not completely. Instead of replacing the house soup everywhere, the kitchen serves the new one to five tables out of fifty and watches. Empty bowls and reorders mean promote it to more tables. Complaints mean pull it, and only five tables ever tasted the mistake.

Tables are requests, the house soup is v1, the new recipe is v2, and the dining room's reaction is your metric window. Promotion to the full menu is the 100 percent rollout.

Where it stops: five tables cannot tell you about Friday's crowd. A tiny slice gives fast warning but weak proof, which is why canaries gate on clear failure signals and let A/B testing settle close calls.

How It Actually Works

A canary runs both models behind one endpoint and grows the new one's share in steps.

Serve both, route mostly to v1. Deploy v2 next to v1 behind the same API. The gateway starts at roughly 90 percent v1 and 10 percent v2, assigned by hashed user id so each user sticks to one model.
Watch two metric families. System metrics cover errors, latency percentiles, and memory. Model metrics cover prediction distributions, confidence, and any fast feedback like clicks. Either family can stop the rollout.
Promote or revert on thresholds. Healthy canary means grow the share on a ladder such as 10 to 25 to 50 to 100 percent, pausing at each rung. A breached threshold reverts routing to 100 percent v1 immediately.

Worked example

Suppose the endpoint handles 20,000 requests during the canary window with a 5 percent slice: 19,000 to v1 and 1,000 to v2. Baseline v1 errors on 20 requests, about 0.11 percent. Canary v2 errors on 15, exactly 1.50 percent, roughly 14 times the baseline rate. That gap dwarfs any sampling wobble at these volumes, so the rollout stops here and v2 goes back for fixes. Had v2 matched v1 at near 0.11 percent with healthy latency, the share would climb the ladder instead. The diagram draws the 90/10 version of this same setup.

Code

Routing is the mechanism, and sticky routing is the version that works. The function below hashes each user into the canary with a fixed probability, so repeated calls for one user always agree.

1import hashlib2
3
4def route(user_id: str, pct: int = 10) -> str:5    """Send pct percent of users to the canary, stickily by user id."""6    bucket = int(hashlib.sha256(user_id.encode()).hexdigest()[:8], 16) % 1007    return "canary-v2" if bucket < pct else "baseline-v1"8
9
10users = ["user-001", "user-002", "user-003", "user-004", "user-005"]11print([route(u) for u in users])12# -> ['baseline-v1', 'baseline-v1', 'canary-v2', 'baseline-v1', 'baseline-v1']13print(route("user-001") == route("user-001"))14# -> True

The outputs show both properties at once: the split spreads users across arms, and the same id routes identically every time.

Watch Out For

One user, two models, one confused session

Splitting per request instead of per user means a chatbot answer changes personality mid-conversation and a recommender forgets what it just showed. Users report ghosts, and your per-arm metrics mix both models' effects into each session.

Hash the user or session id, as the snippet does, and confirm stickiness in the logs: count distinct models per session during the canary. Anything above one per session means the split is wrong, not the model.

A canary too thin to prove anything

At low traffic, 5 percent may mean a few dozen requests, and zero errors in forty requests does not clear a model. Teams then promote on silence and meet the bug at 100 percent, where it was always heading.

Set a minimum sample before each promotion step, not just a percentage. If the rung cannot collect enough requests in a reasonable window, widen the slice or lengthen the window before you trust the green lights.

The Quick Version

A canary serves v2 to a small sticky slice of traffic while v1 carries everyone else.
System metrics and model metrics both gate promotion, and either can trigger instant reversion to v1.
In the worked window, v2 errored at 1.50 percent against v1 at 0.11 percent, so the rollout stopped.
Healthy canaries climb a ladder such as 10 to 25 to 50 to 100 percent instead of jumping.
Route by hashed user id so each person sees one model, and demand enough samples before each step.