Blue-Green Deployments

Two identical production environments take turns serving users. You build and check the new model in the quiet one, then flip the router so all traffic moves at once.

The router flips all live traffic from Blue v1 to Green v2 in one cutover while Blue stands by idle for an instant rollback.

Why Does This Exist?

Replacing a live model in place means stopping the old one to start the new one. Users see errors during the gap, and if the new model misbehaves, recovery means reinstalling the old one under pressure. A bad ten minutes becomes a bad afternoon.

Blue-green removes the gap. You keep two full production environments and never operate on the one serving users. The release becomes a router flip, and the rollback is the same flip in reverse. This page covers that flip and its costs. Gradual traffic shifting instead of one flip is canary deployments, and the serving layer the router points at is model serving.

Basics first: a router is the layer that decides which environment answers each request, usually a load balancer in front of both. Cutover is the moment you change its target. In blue-green, cutover moves 100 percent of traffic in one change, which is what makes both the release and the rollback instant.

Think of It Like This

A stage on a turntable

A theatre must swap its set between acts without keeping the audience waiting. Instead of striking the old set in front of everyone, the stage sits on a giant turntable. The crew builds the second act on the hidden back half while the first act plays on the front half.

At intermission the stage rotates 180 degrees. The new set faces the audience all at once, and the old set sits intact behind the wall. If a flat wobbles in act two, the crew rotates straight back. No rebuilding, no dark stage.

Where it stops: both halves share the theatre's wiring and plumbing. If the new set needs rewiring that the old set cannot use, spinning back does not save you. Shared data stores are that shared wiring, as the pitfalls below explain.

How It Actually Works

Three steps, always in this order: duplicate, verify, flip.

Duplicate the environment. Blue serves 100 percent of traffic while Green idles with identical capacity. Identical means the same model runtime, feature code, and resource shape, pointed at the same data stores.
Deploy and verify in the quiet. Release v2 to Green with no live traffic and run smoke tests, integration checks, and a slice of recorded production requests against it. Nothing users see is touched.
Cut over in one flip. Point the router at Green. Every new request now lands on v2. Keep Blue running untouched until v2 proves itself, so rollback is one flip back.

Worked example

Say v1 on Blue serves 1,200 requests per minute. You deploy v2 to Green and replay 200 recorded requests: zero failures, median latency within 5 percent of Blue. The router flips, and all 1,200 requests per minute move to Green in that one change. An hour later error rate triples, so you flip back to Blue in seconds; users never waited through a reinstall. For contrast, a 5 percent canary of the same traffic would expose only 60 requests per minute to v2. Blue-green buys instant reversal at the price of full duplicate capacity, which the diagram keeps visible as two equal boxes.

Code

The whole mechanism is one variable. The snippet below models the router so you can see the flip and the rollback as plain assignments.

1routes = {"blue": "model-v1", "green": "model-v2"}2active = "blue"3
4
5def serve(request_id: int) -> str:6    """Every request lands on whichever environment is active."""7    return routes[active]8
9
10print([serve(i) for i in range(3)])11# -> ['model-v1', 'model-v1', 'model-v1']12active = "green"  # the cutover: one assignment moves 100 percent13print([serve(i) for i in range(3)])14# -> ['model-v2', 'model-v2', 'model-v2']15active = "blue"  # rollback: the same assignment in reverse16print(serve(99))17# -> model-v1

The outputs confirm the contract: before the flip everything resolves to v1, after it everything resolves to v2, and the old environment answers again the moment you flip back.

Watch Out For

A schema change that rollback cannot undo

The model is rarely what breaks. Both environments share the same feature store and database during the switch, so a v2 migration that renames or drops a column poisons Blue the moment it runs. You flip back to Blue and Blue crashes on the new schema, which turns your instant rollback into a manual repair.

Keep every data change backward compatible across the switch. Add columns before you need them, never rename in the same release that uses the new name, and test rollback itself: flip forward, then back, against the migrated store before you call the release done.

Paying double during the window

Two identical environments means twice the bill while both stand. For a small classifier that is rounding error. For a large model holding 8 GPUs per replica, doubling replicas for a one-hour soak doubles that hour's GPU cost, and a cautious team soaking overnight pays far more than the release itself.

Price the window before you choose the strategy. If duplicate capacity costs more than the outage risk justifies, a canary deployment buys most of the safety for a fraction of the hardware.

The Quick Version

Blue-green keeps two identical production environments and serves users from exactly one at a time.
The new model deploys to the idle environment, where it is verified against real data with zero user impact.
Release is a single router change moving 100 percent of traffic, as in the 1,200 requests per minute example.
Rollback is the same change in reverse, which is why the old environment stays up until the new one proves itself.
Shared data schemas must stay backward compatible or rollback fails, and you pay for double capacity during the window.