A/B Testing

Two models serve live traffic at the same time, and the one that moves the business number wins. You judge them on real user behavior, not offline scores.

Live traffic splits evenly between control model A and challenger model B, and conversions plus a significance check declare the winner.

Why Does This Exist?

Offline scores pick the model that looked best on last month's labels. Production pays you for this week's behavior: clicks, purchases, watch time. Those two verdicts often disagree. A reranker can lift F1 from 0.71 to 0.74 and still move click-through rate not at all, because the offline labels never recorded what users actually wanted next.

So you settle the argument with live evidence. An A/B test runs the current champion and one challenger side by side on real traffic and measures which one moves the business number. This page covers the split, the logging, and the significance call. The shipping mechanics for the winner belong to canary deployments and blue-green deployments.

Basics first: a conversion rate is conversions divided by requests. If 210 of 5,000 served requests convert, the rate is 4.20 percent. A significance check then asks whether the gap between two rates could plausibly be luck. By convention, a p-value below 0.05 means probably not luck. The metrics you guard while the test runs live in model monitoring.

Think of It Like This

Two soup pots at one market stall

You sell soup from a stall and wonder whether a new recipe sells better. You set out two pots, A and B, and serve every other customer from alternating pots. Each pot has its own tip jar, so coins land next to the recipe that earned them. After a weekend you count jars, not compliments.

The mapping is direct. Pots are your two models. Alternating customers is the traffic split. Coins are conversions. Tasting both pots yourself the night before is offline evaluation: informed, but it is not the market deciding.

Where it stops: one stall on one weekend is not the whole city. A rainy-day crowd orders differently from a lunch crowd, so a win with one mix of customers may not repeat. That is why real tests check segments, not just totals.

How It Actually Works

A production A/B test has five moving parts, and skipping any one of them quietly voids the result.

Pick one primary metric plus guardrails. The primary metric is the business number the challenger must move, such as conversion rate. Guardrails are metrics it must not break, such as latency or refund rate. One primary metric keeps the verdict crisp.
Split by user, not by request. Hash the user id and send each user to the same arm for the whole test, usually 50/50 for maximum speed. Per-request coin flips let one person see both models in one session.
Log exposure with outcome. Every served request records which arm served it and what the user did next, joined by a request id. Without that join you have predictions and purchases but no way to connect them.
Size the run before you start. Decide the sample size from the smallest lift worth shipping. Small gaps need long runs: the example below needed 5,000 requests per arm to call a 0.90 point gap.
Compare with a two-proportion check. The standard tool is the z-test on the two conversion rates. Ship the challenger only when the result clears your threshold and every guardrail holds.

Worked example

Both arms get 5,000 requests. Model A converts 210 (4.20 percent) and model B converts 255 (5.10 percent), a gap of 0.90 percentage points. The pooled rate is 465 / 10,000 = 0.0465. The standard error is the square root of 0.0465 times 0.9535 times (1/5000 + 1/5000), which works out to about 0.00421. The z-score is 0.009 / 0.00421, about 2.14, and the two-sided p-value is about 0.0326. That sits below 0.05, so B's lead is unlikely to be luck, and B ships. These are the same numbers drawn in the diagram above.

Code

The snippet below is the check from the worked example, written so it runs anywhere with no extra packages. Feed it the two conversion counts and it returns the verdict.

1import math2
3
4def ab_verdict(x1: int, n1: int, x2: int, n2: int, bar: float = 0.05) -> str:5    """Two-proportion z-test for challenger (x2/n2) against control (x1/n1)."""6    p1, p2 = x1 / n1, x2 / n27    pooled = (x1 + x2) / (n1 + n2)8    se = math.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))9    z = (p2 - p1) / se10    p = 2 * (1 - 0.5 * (1 + math.erf(abs(z) / math.sqrt(2))))11    print(f"A: {p1:.2%}  B: {p2:.2%}  z = {z:.2f}  p = {p:.4f}")12    # -> A: 4.20%  B: 5.10%  z = 2.14  p = 0.032613    return "ship B" if p < bar else "keep A"14
15
16print(ab_verdict(210, 5000, 255, 5000))17# -> ship B

It prints the same 4.20 percent, 5.10 percent, z of 2.14 and p of 0.0326 derived by hand above. Change the counts to your arms and the same function gives your verdict.

Watch Out For

A segment mix that flips the verdict

B can win overall while losing in every segment. Suppose mobile users convert at half the desktop rate and arm B accidentally receives more desktop traffic. B's total looks better even though A beats it on mobile and on desktop separately. The symptom is a winner whose edge vanishes the moment you slice by device, region, or traffic source.

Fix it at design time. Split by hashed user id so segments balance, and always re-cut the result by your two or three biggest segments before you ship. If the arms differ in mix, the totals mean nothing.

Stopping the moment B looks ahead

Peeking breaks the math. If you check significance every morning and stop the first day B clears 0.05, your false-positive rate is far above 5 percent. Fourteen daily peeks at a dead-even test produce a spurious winner roughly twice as often as the nominal rate suggests.

Fix it by pre-registering the run. Compute the sample size for the smallest lift worth shipping, then run to that size without deciding early. If you must look, use sequential methods built for looking instead of a fixed-sample test.

The Quick Version

An A/B test serves the champion and one challenger on live traffic at the same time and compares them on a business metric.
Split users 50/50 by hashed id so each person sees one model consistently through the test.
Log which arm served each request and join it to what the user did next.
Size the run up front; a 0.90 point lift at 5,000 requests per arm gives p near 0.03.
Ship only on significance plus intact guardrails, and confirm the win inside each major segment.