Skip to content
AI360XpertBeta

Randomized Experiments

When you flip a coin to decide who gets a treatment, you guarantee that nothing else caused the decision. Randomization severs the link from all confounders, known and unknown.

A coin flip cuts the Z to T arrow, so a biased plus 10.15 point guess falls to plus 2.07 near the true plus 2
A coin flip cuts the Z to T arrow, so a biased plus 10.15 point guess falls to plus 2.07 near the true plus 2

Why Does This Exist?

Observational data lets you adjust for confounders you measured. If income and age shape both the email and the purchase, add them to the model. The trouble is the confounders you never measured: boredom, a 3 AM inbox habit, a hidden preference. No math removes bias from a variable that is not in the data.

Randomized experiments exist to remove that whole class of bias at the design stage. Hand assignment to a coin flip and treatment becomes independent of every other variable, measured or not. That is why an online test in A/B testing for ML can claim causation while a regression on past logs cannot.

This page covers the coin flip fix. The language of alternate universes behind the effect lives in potential outcomes. The graph surgery that deletes the arrow lives in causal graphs. The weighting repair when you cannot randomize lives in propensity score methods.

Think of It Like This

Wetsuits, coin flips and swim times

You watch a lake and see wetsuit swimmers finishing far ahead of swimsuit swimmers. Wetsuits look fast. But wetsuit swimmers are mostly triathletes and swimsuit swimmers are mostly weekend visitors. Athleticism is the hidden confounder, and the suit gets credit for the swimmer.

Now take one mixed group and flip a coin for each person. Heads gets a wetsuit, tails gets a swimsuit. Some triathletes land in swimsuits and some beginners land in wetsuits. The coin does not know fitness, so fitness spreads evenly on average. The remaining gap in finish times must come from the suit.

Each swimmer is a student, the suit is the study tool, athleticism is hidden motivation, the coin is random assignment, and the finish gap is the average treatment effect.

Where it stops: swimmers feel the suit and may try harder, and a coin cannot force effort. People also refuse arm bands they are assigned. Assignment is not always receipt, which is why non-compliance needs its own fix below.

How It Actually Works

The effect you want

Call treatment TT, with T=1T = 1 for tool and T=0T = 0 for no tool. Call outcome YY, the exam score. Write Y(1)Y(1) for the score if treated and Y(0)Y(0) for the score if not. The average treatment effect is the gap between those two alternate universes:

ATE=E[Y(1)]−E[Y(0)]\text{ATE} = \mathbb{E}[Y(1)] - \mathbb{E}[Y(0)]

You only ever see one universe per person. The simple gap in observed means, E[Y∣T=1]−E[Y∣T=0]\mathbb{E}[Y \mid T = 1] - \mathbb{E}[Y \mid T = 0], equals the ATE only when treatment is independent of those alternate outcomes. That independence is called exchangeability. Randomization creates it by construction.

The arrow randomization deletes

Draw confounder ZZ pointing at both TT and YY, so T←Z→YT \leftarrow Z \rightarrow Y. That fork is the backdoor path. Normally some rule sets TT from ZZ: motivated students pick the tool. Randomization replaces that rule with T=coin flipT = \text{coin flip}. Treatment now listens only to the coin, so the arrow Z→TZ \rightarrow T is deleted. The backdoor path is cut in the graph, not adjusted in a model.

Worked example: 5000 students, one hidden motive

Baseline is 70 points. Hidden motivation adds five points per unit. The tool adds exactly 2.0 points. Motivation is unmeasured, so the observational regression cannot include it.

In observational data, motivated students pick the tool, so the treated group is already stronger. The naive slope lands near plus 10.15 points, crediting the tool for motivation. In randomized data, half get the tool by coin flip, so motivation spreads evenly. The same naive slope lands near plus 2.07 points, close to the true plus 2. Same formula, same N=5000N = 5000. The only change is who decides TT.

Large NN matters because the coin balances traits on average. With 20 students a flip can still stack one side with older or wealthier users by luck. Randomization guarantees balance in expectation, not in every small draw.

Code

The point is the check: one hidden confounder breaks the observational slope and leaves the randomized slope near the truth.

import numpy as npimport statsmodels.api as sm
np.random.seed(42)N: int = 5000
hidden_motivation: np.ndarray = np.random.normal(0, 1, N)
true_effect: float = 2.0Y_0: np.ndarray = 70 + 5 * hidden_motivation + np.random.normal(0, 2, N)Y_1: np.ndarray = Y_0 + true_effect
prob_T_obs: np.ndarray = 1 / (1 + np.exp(-2 * hidden_motivation))T_obs: np.ndarray = np.random.binomial(1, prob_T_obs)Y_obs: np.ndarray = np.where(T_obs == 1, Y_1, Y_0)
X_obs: np.ndarray = sm.add_constant(T_obs)model_obs = sm.OLS(Y_obs, X_obs).fit()print(f"Observational Estimate: +{model_obs.params[1]:.2f} points")# -> Observational Estimate: +10.15 points
T_rand: np.ndarray = np.random.binomial(1, 0.5, N)Y_rand: np.ndarray = np.where(T_rand == 1, Y_1, Y_0)
X_rand: np.ndarray = sm.add_constant(T_rand)model_rand = sm.OLS(Y_rand, X_rand).fit()print(f"Randomized Estimate: +{model_rand.params[1]:.2f} points")# -> Randomized Estimate: +2.07 points

The printout matches the hand story: observational near plus 10.15 because motivation rides along, randomized near plus 2.07 because the coin spread motivation evenly. Flip the true effect to 0 and the randomized slope falls near 0 while the observational slope stays high.

Watch Out For

Trusting a tiny randomized sample

Randomization balances traits on average, not in every small draw. Flip a coin ten times and eight heads is ordinary.

Symptom: a 20 person pilot shows a huge lift that vanishes on rollout, and the treatment group was older or richer by luck. Fix: pre-register a larger NN, check balance on age and prior behavior before reading the lift, and re-randomize when a check fails. Small samples bring confounding back through variance.

Reading assignment as receipt

Random assignment is not random use. Some treated users never open the feature and some control users find it anyway.

Symptom: the gap between assigned groups looks small, so a team ships or kills a feature on a diluted number. Fix: report the intention to treat effect for the assignment, then instrument for actual use when the receipt question matters. See A/B testing for ML for the guardrails that keep the assignment clean.

The Quick Version

  • Randomized experiments assign treatment by coin flip, which deletes the arrow from every confounder to treatment.
  • The target is the average treatment effect, the mean gap between the treated universe and the untreated universe.
  • With 5000 students and a true plus 2 point tool, the observational slope reads near plus 10.15 because motivation picks the tool.
  • The same slope on coin flip data reads near plus 2.07 because motivation spreads evenly across arms.
  • Exchangeability means treated and control groups are interchangeable, so the observed gap equals the causal gap.
  • Small samples can still unbalance by luck, and assigned treatment is not always taken treatment.