A/B Testing for ML
Split your users at random, give half the old model and half the new one, then compare what each group actually does. If the new group buys more, the model caused it.
Why Does This Exist?
Offline scores lie by omission. A recommender can post a strong F1 while pushing items people would have bought anyway, adding nothing to revenue. You meet this gap the day a model graduates from a held-out set to live traffic: the number that picked the winner stops describing what users actually do.
An A/B test closes that gap. You split users at random, serve the old system to one half and the new model to the other, and compare a business number like conversion or revenue per user. Randomness does the heavy lifting here. It balances everything you cannot see, seasonality, device mix, user mood, so a gap you can measure is one the model caused. That setup is a randomized controlled trial, the same design medicine uses, pointed at your model.
This page covers the online experiment itself: how to randomize, what to measure, and how to tell a real lift from noise. Offline scoring, the part that happens before any user sees the model, lives in model evaluation. What to do when you cannot randomize at all lives in propensity score methods.
One piece of grounding before the mechanism. A conversion rate is conversions divided by users. A rate over 10,000 users barely wobbles from day to day, while a rate over 100 swings wildly. Sample size decides everything below, which is why this page keeps repeating its counts.
Think of It Like This
Two tomato beds, one fertilizer
You invent a fertilizer and want to know if it works. You mark off two beds in the same field, spread the new mix on one, and leave the other alone. Same sun, same rain, same soil underneath. At harvest you weigh both beds. If the fed bed is heavier, the fertilizer did it, because weather and soil hit both sides equally.
Your users are the beds. The old model is the unfed bed, the new model is the fed one, and random assignment is what keeps both beds in the same field. The harvest weights are your two conversion rates.
Where it stops: tomato beds never swap sides, never hear about the other bed, and never get bored of fertilizer. Your users do all three. They share accounts, talk to each other, and react to novelty, and each of those breaks the clean-bed picture. The pitfalls at the end of this page are exactly those breakages.
How It Actually Works
Randomize users, not page views
Randomize the wrong unit and the test quietly dies. If you flip a coin per page view, one person sees the old model on Monday and the new one on Tuesday, and neither experience is real. So you hash the user ID into a bucket once, and that person stays in their arm for the whole test. A 50/50 split is the default; you tilt traffic only when the new model is risky enough that a full half feels reckless.
Measure money, guard the rest
Offline ML numbers stay home for this test. You judge it on one primary metric, the overall evaluation criterion: conversion rate, revenue per user, retention, whatever the model is for. Then you set guardrails, the numbers you refuse to break: latency, crash rate, unsubscribe rate. A test that lifts revenue while doubling latency is a failure wearing a win costume. Ship needs both sides green.
Decide with a test, not your gut
Call the control rate (plain words: share of control users who converted) and the treatment rate (same share in the new-model arm), with and users in each arm. The null hypothesis says the model does nothing, so any gap is luck. The alternative says the gap is real.
The observed lift is the difference:
But a difference means nothing without its wobble. The standard error measures that wobble: how much the gap would bounce around if you reran the test with the same true rates:
Divide signal by wobble to get the z-score , the gap measured in units of its own noise. Past you call it significant at the 5 pct level:
The p-value is the same idea in probability clothes: if the model truly did nothing, how often would luck alone produce a gap this big? Here the answer is about 0.058, roughly a 1-in-17 fluke. Interesting is not significant. You do not ship on it.
Worked example: 5.0 pct against 5.6 pct
Make it concrete. You run 10,000 users per arm. Control converts 500 of 10,000, which is 5.0 pct. Treatment converts 560 of 10,000, which is 5.6 pct. Those are the numbers drawn in the diagram above.
The absolute lift is , six tenths of a point. The relative lift is , a 12 pct jump that feels shippable. But the wobble is , so falls short of 1.96. A 12 pct relative lift with 10,000 users per arm still is not significant at the 5 pct level, and that surprise is the whole lesson: eyeballing percentages without the denominator ships noise. Run longer or accept the smaller detectable effect your traffic allows.
Code
This snippet replays the worked example with the standard library only. It computes the lift and the z-score from raw counts, then checks them against the 1.96 bar for 5 pct significance.
import math
def two_proportion_z(x_c: int, n_c: int, x_t: int, n_t: int) -> tuple[float, float]: """Return (absolute lift, z-score) for treatment against control.""" p_c = x_c / n_c # control conversion rate p_t = x_t / n_t # treatment conversion rate se = math.sqrt(p_c * (1 - p_c) / n_c + p_t * (1 - p_t) / n_t) lift = p_t - p_c return lift, lift / se
lift, z = two_proportion_z(500, 10_000, 560, 10_000)print(f"{lift:.4f}") # -> 0.0060print(f"{z:.2f}") # -> 1.89print(abs(z) > 1.96) # -> FalseThe printout matches the hand computation: lift 0.0060, z 1.89, below the bar. False is the test telling you to keep running, not to ship. For large samples like these the unpooled standard error above is the standard choice; with tiny samples you would reach for an exact test instead.
Watch Out For
Peeking daily, then shipping a mirage
The symptom: you check the p-value every morning, stop the day it dips under 0.05, ship, and watch the lift evaporate. Peeking like that can push your false positive rate several multiples above the 5 pct you think you chose, because you gave luck many chances to fool you and kept the first success.
Fix it before launch, not after. Pre-register the runtime or the sample size and do not look until it is done. If you truly need early stopping, use a sequential method that spends its error budget across looks instead of an ordinary test. Never steer by a daily p-value chart.
One arm leaking into the other
The symptom: you test a pricing model on a marketplace, treatment sellers undercut control sellers, control demand drains away, and both arms drift together. The test assumed each user's outcome depends only on their own assignment, and your marketplace violates it. That assumption has a name, SUTVA, and interference breaks it.
Fix it by randomizing bigger chunks. Assign whole cities or time windows instead of users, so the spillover stays inside the chunk; that design is called a switchback or geo experiment. And when users share the thing being ranked, read confounding and colliders before you trust any per-user split. If the goal is adapting traffic mid-test rather than a clean verdict, the sequel is contextual bandits.
The Quick Version
- An A/B test is a randomized controlled trial on your traffic: random halves, old system on one side, new model on the other.
- Randomize by hashed user ID so each person gets one consistent experience for the whole test, never by page view.
- Judge the test on one business number like conversion, with guardrails on latency, crashes, and churn that can veto any win.
- A 0.60-point gap on 10,000 users per arm gives a z-score near 1.89, short of the 1.96 bar, so you keep running instead of shipping.
- Checking the p-value daily and stopping at the first green reading manufactures winners; fix the runtime up front or use a sequential method.
- When users in one arm affect the other, the test is void; randomize whole cities or time windows instead.