Sequential Testing

Why Does This Exist?

The golden rule of traditional A/B testing is no peeking. You must calculate your required sample size in advance, run the test for exactly that duration, and only check the p-value at the very end.

If you peek at the results every day and stop the test the moment the p-value drops below 0.05, you will massively inflate your False Positive Rate (Type I error). A test designed to have a 5% false positive rate might actually have a 30% false positive rate if you peek daily, because random noise will almost always briefly dip the p-value below 0.05 at some point during the run.

But in industry, waiting two weeks for a test that is obviously winning (or obviously destroying revenue) is painful. Sequential Testing exists to solve this. It is a family of statistical methods that adjust the math so you can legitimately peek at the data continuously, allowing you to stop successful experiments early and kill bad ones immediately.

Think of It Like This

How It Actually Works

Sequential testing abandons the fixed-sample-size assumption of classical frequentist statistics (like the standard t-test) and uses dynamic boundaries or entirely different probability frameworks.

Always-Valid P-Values

The most common modern approach uses Always-Valid P-Values (often built on mixture martingales). Instead of a static threshold (like 0.05), the boundary for significance changes as more data comes in.

• At 1,000 users, the evidence required to stop the test is massive.
• At 100,000 users, the evidence required is much smaller.

This dynamic boundary guarantees that no matter how many times you peek, your overall False Positive Rate over the lifetime of the experiment will never exceed your target (e.g., $\alpha = 0.05$).

The Sequential Probability Ratio Test (SPRT)

Developed by Abraham Wald during World War II for quality control, SPRT keeps a running tally of the log-likelihood ratio between two hypotheses. Instead of waiting for $N$ samples, it updates a score after every single observation. The test has two thresholds: an upper boundary (Accept Treatment) and a lower boundary (Reject Treatment). The test continues until the score crosses one of the boundaries. It is mathematically guaranteed to reach a decision faster, on average, than a fixed-horizon test.

Show Me the Code

This code demonstrates the "Peeking Problem" with a standard t-test, showing how continuous monitoring creates false positives even when the models are identical (no real lift).

1import numpy as np2import scipy.stats as stats3import matplotlib.pyplot as plt4
5np.random.seed(42)6days = 147users_per_day = 10008
9# True Conversion Rates are IDENTICAL (5.0%). The new model does nothing.10true_c = 0.05011true_t = 0.05012
13cumulative_control = []14cumulative_treatment = []15p_values = []16
17# Simulate the test running for 14 days18for day in range(1, days + 1):19    # Daily new traffic20    daily_control = np.random.binomial(1, true_c, users_per_day)21    daily_treatment = np.random.binomial(1, true_t, users_per_day)22    23    cumulative_control.extend(daily_control)24    cumulative_treatment.extend(daily_treatment)25    26    # We "Peek" every day by running a t-test on the cumulative data27    t_stat, p_val = stats.ttest_ind(cumulative_treatment, cumulative_control)28    p_values.append(p_val)29    30    if p_val < 0.05:31        print(f"Day {day}: p={p_val:.3f}. FALSE POSITIVE! The test 'won' due to noise.")32        # If we didn't know better, we would stop the test here and deploy.33
34# Look at the final day's result (The only valid time to check in traditional A/B testing)35print(f"Final Day p-value: {p_values[-1]:.3f}. (Correctly not significant).")

If you run a simulation like this 1,000 times, you will find that continuous peeking causes the test to falsely declare a winner roughly 20-30% of the time, instead of the promised 5%. Sequential testing adjusts the 0.05 threshold dynamically so this inflation doesn't happen.

Watch Out For

The Quick Version

• The Peeking Problem: Checking a standard A/B test daily and stopping early inflates your false positive rate, causing you to deploy models that don't actually work.
• Sequential Testing mathematically corrects for continuous monitoring.
• It uses dynamic decision boundaries (like Always-Valid P-Values or SPRT) that allow you to stop the test early if the effect is massive, without increasing your error rate.
• It is the standard evaluation method for modern experimentation platforms at companies like Netflix, Uber, and Optimizely.

What to Read Next

• ab-testing-for-ml — Review the standard fixed-horizon A/B testing framework.
• online-evaluation — Other ways to continuously monitor models safely in production.
• ab-testing-models — Architecture patterns for routing traffic during an A/B test.