Statistics and A/B Testing

Hypothesis tests, confidence intervals, power, sample size, variance reduction, and the traps that break experiments.

Experiment Workflow

Hypothesis + metric
Power + sample size
Randomize
Run full duration
Check guardrails + SRM
Analyze + decide

Decide the metric, MDE and duration before launch

Primary metric: One pre-registered decision metric per experiment.
Guardrail metrics: Latency, errors, revenue, or churn that must not get worse.
Unit of randomization: Usually the user; must match the unit of analysis.

Hypotheses and Errors

	H₀ true	H₀ false
Reject H₀	Type I error (α)	Correct (power = 1 − β)
Keep H₀	Correct	Type II error (β)

Null H₀: No difference between control and treatment.
Alternative H₁: A difference exists (one- or two-sided).
α: Tolerated false-positive rate, commonly 0.05.

P-values

A p-value measures how surprising the data would be if there were no real difference.

Definition: Probability of data at least this extreme if H₀ were true.
Not: The probability that H₀ is true, or the size of the effect.
Decision: Reject H₀ when p < α; report the effect and its interval too.
Practical significance: A tiny effect can be significant with huge samples.

p = 0.04 is not "96% sure"

A p-value says how surprising the data is under the null, not how likely the treatment works.

Standard Error and Confidence Intervals

SE(x̄) = s / √n

SE(p̂) = √(p̂(1 − p̂) / n)

CI = estimate ± z₁₋α/₂ · SE

z = 1.96 for a 95% interval

SE(Δ) = √(s₁²/n₁ + s₂²/n₂)

Interpretation: The procedure covers the true value in 95% of repeated experiments.
Width: Shrinks with √n; quadrupling the sample halves the interval.

Choosing a Test

Data	Test
Two means, large n	Welch's t-test (z-test at scale)
Two proportions	Two-proportion z-test or chi-square
Skewed or heavy-tailed	Mann–Whitney U or bootstrap
Paired measurements	Paired t-test
Many groups	ANOVA, then corrected pairwise tests
Ratio metrics	Delta method or bootstrap

Effect Size and Power

d = (μ₁ − μ₂) / σ

Cohen's d: 0.2 small, 0.5 medium, 0.8 large

Power: Probability of detecting a true effect of a given size; target 80%.
Drivers: Larger effect, larger sample, lower variance, or higher α raise power.
MDE: Minimum detectable effect, the smallest lift the test can reliably detect.

Sample Size

n per group ≈ 2(z₁₋α/₂ + z₁₋β)² σ² / δ²

n per group ≈ 16 σ² / δ²

second line: rule of thumb for α = 0.05, power = 80%

Proportions: Use σ² = p(1 − p) at the baseline rate.
Halving the MDE: Requires about four times the sample.
Duration: Run whole weeks to cover weekday and weekend behavior.

Compute the sample size before launch, then commit to it. Stopping when it "looks significant" invalidates the test.

Randomization Checks

Sample ratio mismatch (SRM): Observed split differs from the design; chi-square test on counts.
A/A test: Two identical arms should differ significantly only about α of the time.
Pre-period balance: Covariates should look similar across arms before exposure.
Exposure logging: Log assignment at the moment the user sees the change.

SRM means stop

A sample ratio mismatch points to a bug in assignment or logging. Fix it before reading any results.

Multiple Testing

Problem: With 20 independent tests at α = 0.05, expect about one false positive.
Bonferroni: Use α / m per test; simple and conservative.
Holm: Step-down version of Bonferroni with more power.
Benjamini–Hochberg: Controls the false discovery rate; good for many metrics.
Pre-register: Name the primary metric so extra metrics stay exploratory.

Sequential Testing and Peeking

Peeking: Checking repeatedly and stopping at the first p < α inflates false positives.
Fix 1: Fixed horizon: analyze once at the planned sample size.
Fix 2: Sequential designs (group sequential, always-valid p-values, mSPRT).
Early stopping for harm: Guardrail breaches can stop a test with pre-agreed rules.

Variance Reduction

Y_cuped = Y − θ(X − mean(X)), θ = Cov(X, Y) / Var(X)

X = the same metric measured before the experiment

CUPED: Uses pre-experiment data to remove predictable variance.
Stratification: Randomize or analyze within strata such as country or platform.
Winsorizing: Cap extreme values of heavy-tailed metrics like revenue.
Better metrics: Binary or capped metrics usually have lower variance.

Ratio and Clustered Metrics

Ratio metric: For example clicks per session, where the denominator is random too.
Delta method: Approximates the variance of a ratio of means.
Cluster effects: Randomizing by user but analyzing per page view underestimates variance.
Fix: Analyze at the randomization unit, or use cluster-robust standard errors.

Bayesian A/B Testing

p ~ Beta(α + successes, β + failures)

Output: P(treatment > control) and the expected loss of choosing each arm.
Priors: Weakly informative priors keep early estimates sane.
Caution: Bayesian results still suffer when you peek and stop on a threshold without a plan.

Common Experiment Metrics

Metric	Type	Notes
Conversion rate	Proportion	Binomial variance
Revenue per user	Mean, heavy tail	Winsorize or bootstrap
Click-through rate	Ratio	Delta method
Retention (day N)	Proportion	Needs full follow-up window
Latency p95	Quantile	Bootstrap the quantile

Failure Modes

Novelty effect: Early lift fades as users get used to the change.
Interference: Treated users affect control users (marketplaces, social networks).
Simpson's paradox: An aggregate trend reverses inside every subgroup.
Survivorship: Analyzing only users who completed a step biases the comparison.
Seasonality: Holidays or launches distort short tests.

Decision Checklist

The hypothesis, primary metric, and MDE were written before launch.
The sample size and duration were met, with no early stopping.
No sample ratio mismatch; A/A checks pass.
Guardrail metrics did not degrade.
The effect, its CI, and practical significance are reported.
The result is consistent across key segments, or the differences are explained.