Propensity Score Methods
When a trial is impossible, squeeze age, income, and history into one number, then compare treated and control users who share it.
Why Does This Exist?
A team ships a new coaching email and wants its effect on spending. It cannot force half the users to receive it and half to miss it. The logs show buyers who got the email spend more, but those users were already younger, richer, and more active. The raw contrast mixes the email with everything that selected who got it.
One idea first. A confounder drives both treatment and outcome , so and move together even when does nothing. Causal graphs draw those arrows, and confounding and colliders show which junctions to adjust for and which to leave alone. This page covers the adjustment that follows: compress into one number and rebalance on it. The clean fix when you can randomize lives in randomized experiments. The counterfactual framing lives in potential outcomes.
Think of It Like This
Twin day at a running club
A coach wants to know whether a new shoe makes runners faster. Runners who bought it already train more, so a straight comparison of wearers against everyone else rewards fitness, not foam.
So the coach holds a twin day. A 30 year old who runs 40 km a week and wears the shoe is paired with another 30 year old who runs 40 km a week and does not. Same age, same mileage, same route, one difference. Repeat for every wearer, average the pair gaps, and fitness cancels out.
Each pair is a match on background traits, the pile of traits is the confounder set, and the stopwatch gap inside a pair is the effect. The propensity score is the shortcut that finds the twin without matching on every trait separately.
Where it stops: twins share what the coach wrote down. If one twin sleeps eight hours and the other sleeps five, and sleep was never recorded, the pair still differs. Matching balances only measured traits, never hidden ones.
How It Actually Works
The score that compresses the background
Write for the propensity score, meaning the chance of treatment given traits . It is one number in . Fit it with any classifier, often logistic regression, with as inputs and as the label. Age, income, and past activity go in. Anything the treatment causes stays out, because adjusting for a consequence manufactures bias.
The balancing claim is direct: users with the same have the same mix of on average across arms, so comparing inside a score level compares like with like.
Check overlap before estimating
If treated scores pile near 0.99 while control scores stop at 0.40, some treated users have no twin. That region has no common support, meaning no overlap, and it must be dropped or the estimand changes to the overlap population. Plot both histograms first. No overlap, no match.
Worked example: four users, naive 5.0, weighted 6.8
Take two treated and two control users. Write each as outcome with score:
- A, treated, outcome 10, score 0.8
- B, treated, outcome 14, score 0.2
- C, control, outcome 6, score 0.8
- D, control, outcome 8, score 0.2
Naive means are treated and control, so the naive gap is . It mixes the email with the traits that predicted it.
Inverse probability weighting builds a pseudo population where treatment looks randomized. Treated weights are and control weights are :
Rare users count most: B took treatment despite 0.2, and C skipped it despite 0.8. Weighted treated mean is . Weighted control mean is . The weighted gap is . Same four rows, different answer, because the weights stand in for the trial that never ran. Matching tells the same story pair by pair: A against C gives at 0.8, B against D gives at 0.2.
Code
The point is the check: rebuild the naive gap 5.0 and the weighted gap 6.8 from the four rows above.
treated: list[tuple[float, float]] = [(10, 0.8), (14, 0.2)]control: list[tuple[float, float]] = [(6, 0.8), (8, 0.2)]
def mean(xs: list[float]) -> float: return sum(xs) / len(xs)
naive_t: float = mean([v for v, _ in treated])naive_c: float = mean([v for v, _ in control])print(f"{naive_t:.1f} {naive_c:.1f} {naive_t - naive_c:.1f}") # -> 12.0 7.0 5.0
wt: list[float] = [1 / p for _, p in treated]wc: list[float] = [1 / (1 - p) for _, p in control]print(f"{wt[0]:.2f} {wt[1]:.2f} {wc[0]:.2f} {wc[1]:.2f}") # -> 1.25 5.00 5.00 1.25
adj_t: float = sum(v * w for (v, _), w in zip(treated, wt)) / sum(wt)adj_c: float = sum(v * w for (v, _), w in zip(control, wc)) / sum(wc)print(f"{adj_t:.1f} {adj_c:.1f} {adj_t - adj_c:.1f}") # -> 13.2 6.4 6.8The printout matches the hand arithmetic. Move B outcome from 14 to 10 and watch the weighted gap fall from 6.8 to 3.6, which shows how much rides on one rare row.
Watch Out For
A confounder that was never measured
Scores balance age and income perfectly, overlap looks clean, and the estimate still misses by a mile.
Symptom: matched pairs differ on something the table never held, like motivation or night shift work, and the gap moves whenever a new column arrives. Fix: name the adjustment set from a graph before fitting, never from column availability, and say plainly what would break the result. Scores cannot balance what was never recorded. An instrumental variable design or a real trial is the sequel when the missing trait matters.
Wild weights from scores near zero or one
One user scores 0.99 or 0.01 and the weight reads 100. The estimate swings with every rerun and confidence intervals widen past usefulness.
Symptom: a handful of rows own most of the weight sum, and dropping three rows moves the gap by points. Fix: trim scores outside 0.05 to 0.95 or stabilize weights by their mean, then report the overlap population and the trim rate. If trimming drops a tenth of treated users, the question changed, so state the new one.
The Quick Version
- The propensity score compresses background traits into one treatment probability in .
- Matching compares treated and control users at the same score, while weighting rebalances the sample with and .
- Our four users at 0.8 and 0.2 give naive means 12.0 and 7.0 for gap 5.0.
- Weights 1.25 and 5.0 give adjusted means 13.2 and 6.4 for gap 6.8.
- No overlap means no twin, so drop the region and rename the population.
- Unmeasured traits and extreme weights break the estimate, so graph the adjustment set and trim near zero or one.