A/B Testing for ML

Why Does This Exist?

When you train a machine learning model, you evaluate it offline using metrics like RMSE, F1-score, or AUC. But high offline accuracy does not guarantee business impact. A recommendation model might have a perfect F1-score because it recommends popular items everyone buys anyway, providing zero incremental lift.

A/B testing exists to bridge the gap between model accuracy and business value. By randomly assigning half of your users to experience the old system (Control) and half to experience the new ML model (Treatment), you create a Randomized Controlled Trial (RCT). This physical randomization severs all confounding variables, proving that any difference in revenue, click-through rate, or retention was definitively caused by the new model.

Think of It Like This

How It Actually Works

A standard A/B test has three core components: the randomization unit, the metrics, and the statistical test.

1. The Randomization Unit

You must decide what to randomize. If you randomize by page view, the same user might see the Control model on Monday and the Treatment model on Tuesday, ruining the experiment. In ML, we usually randomize by User ID (hashing the ID to assign them to a bucket) to ensure a consistent user experience.

2. Guardrail and Primary Metrics

Offline ML metrics (like Log Loss) are useless in an A/B test. Instead, we define:

• Primary Metric (OEC - Overall Evaluation Criterion): What are we trying to improve? (e.g., Conversion Rate, Revenue per User).
• Guardrail Metrics: What are we afraid of breaking? (e.g., Latency, App Crashes, Unsubscribe Rate). A test that increases revenue but quadruples latency is a failure.

3. Hypothesis Testing

Once the test concludes, we have two means: $\mu_{control}$ and $\mu_{treatment}$. We use a statistical test (like a two-sample t-test) to calculate a p-value. The p-value answers the question: If the new model actually had zero effect, how likely is it that we would see a difference this large just by random noise? If $p < 0.05$, we reject the null hypothesis and declare the model a winner.

Show Me the Code

Here is a simplified simulation of evaluating an A/B test using a two-sample t-test to check if the new ML model significantly improved the conversion rate.

1import numpy as np2import scipy.stats as stats3
4np.random.seed(42)5
6# Simulate 10,000 users in each group7n_control = 100008n_treatment = 100009
10# Control group: Baseline conversion rate is 5.0%11# 1 = Converted, 0 = Did not convert12control_conversions = np.random.binomial(1, 0.050, n_control)13
14# Treatment group: New ML model conversion rate is 5.6%15treatment_conversions = np.random.binomial(1, 0.056, n_treatment)16
17# Calculate means18mean_c = np.mean(control_conversions)19mean_t = np.mean(treatment_conversions)20absolute_lift = mean_t - mean_c21relative_lift = (absolute_lift / mean_c) * 10022
23print(f"Control Conversion:   {mean_c:.4f}")24print(f"Treatment Conversion: {mean_t:.4f}")25print(f"Relative Lift:        +{relative_lift:.2f}%")26
27# Perform a two-sample t-test28t_stat, p_value = stats.ttest_ind(treatment_conversions, control_conversions)29
30print(f"\nT-statistic: {t_stat:.2f}")31print(f"P-value:     {p_value:.5f}")32
33if p_value < 0.05:34    print("Result: STATISTICALLY SIGNIFICANT. Deploy the new ML model!")35else:36    print("Result: NOT SIGNIFICANT. The lift might just be noise.")

Watch Out For

The Quick Version

• A/B Testing is a Randomized Controlled Trial applied to software and ML models.
• It proves causal business impact, bridging the gap between offline accuracy and online value.
• You must randomize carefully (usually by User ID) and measure business metrics, not ML metrics.
• Statistical significance guarantees that the observed lift wasn't just random noise, provided you don't peek at the results early.

What to Read Next

• sequential-testing — How to legally "peek" at your A/B test results and stop early without ruining your math.
• ab-testing-models — Specialized routing patterns for deploying ML models into A/B tests (e.g., Shadow deployments).
• online-evaluation — How to continuously monitor models in production after the A/B test ends.