Skip to content
AI360XpertBeta

Propensity Score Methods

When a trial is impossible, squeeze age, income, and history into one number, then compare treated and control users who share it.

Four users at scores 0.8 and 0.2 carry weights 1.25 and 5.0, so the naive gap 5.0 becomes the weighted gap 6.8
Four users at scores 0.8 and 0.2 carry weights 1.25 and 5.0, so the naive gap 5.0 becomes the weighted gap 6.8

Why Does This Exist?

A team ships a new coaching email and wants its effect on spending. It cannot force half the users to receive it and half to miss it. The logs show buyers who got the email spend more, but those users were already younger, richer, and more active. The raw contrast mixes the email with everything that selected who got it.

One idea first. A confounder XX drives both treatment TT and outcome YY, so TT and YY move together even when TT does nothing. Causal graphs draw those arrows, and confounding and colliders show which junctions to adjust for and which to leave alone. This page covers the adjustment that follows: compress XX into one number and rebalance on it. The clean fix when you can randomize lives in randomized experiments. The counterfactual framing lives in potential outcomes.

Think of It Like This

Twin day at a running club

A coach wants to know whether a new shoe makes runners faster. Runners who bought it already train more, so a straight comparison of wearers against everyone else rewards fitness, not foam.

So the coach holds a twin day. A 30 year old who runs 40 km a week and wears the shoe is paired with another 30 year old who runs 40 km a week and does not. Same age, same mileage, same route, one difference. Repeat for every wearer, average the pair gaps, and fitness cancels out.

Each pair is a match on background traits, the pile of traits is the confounder set, and the stopwatch gap inside a pair is the effect. The propensity score is the shortcut that finds the twin without matching on every trait separately.

Where it stops: twins share what the coach wrote down. If one twin sleeps eight hours and the other sleeps five, and sleep was never recorded, the pair still differs. Matching balances only measured traits, never hidden ones.

How It Actually Works

The score that compresses the background

Write e(X)=P(T=1∣X)e(X) = P(T = 1 \mid X) for the propensity score, meaning the chance of treatment given traits XX. It is one number in [0,1][0, 1]. Fit it with any classifier, often logistic regression, with XX as inputs and TT as the label. Age, income, and past activity go in. Anything the treatment causes stays out, because adjusting for a consequence manufactures bias.

The balancing claim is direct: users with the same e(X)e(X) have the same mix of XX on average across arms, so comparing inside a score level compares like with like.

Check overlap before estimating

If treated scores pile near 0.99 while control scores stop at 0.40, some treated users have no twin. That region has no common support, meaning no overlap, and it must be dropped or the estimand changes to the overlap population. Plot both histograms first. No overlap, no match.

Worked example: four users, naive 5.0, weighted 6.8

Take two treated and two control users. Write each as outcome with score:

  • A, treated, outcome 10, score 0.8
  • B, treated, outcome 14, score 0.2
  • C, control, outcome 6, score 0.8
  • D, control, outcome 8, score 0.2

Naive means are (10+14)/2=12.0(10 + 14) / 2 = 12.0 treated and (6+8)/2=7.0(6 + 8) / 2 = 7.0 control, so the naive gap is 12.0−7.0=5.012.0 - 7.0 = 5.0. It mixes the email with the traits that predicted it.

Inverse probability weighting builds a pseudo population where treatment looks randomized. Treated weights are 1/e(X)1 / e(X) and control weights are 1/(1−e(X))1 / (1 - e(X)):

wA=1/0.8=1.25wB=1/0.2=5.0w_A = 1 / 0.8 = 1.25 \quad w_B = 1 / 0.2 = 5.0 wC=1/(1−0.8)=5.0wD=1/(1−0.2)=1.25w_C = 1 / (1 - 0.8) = 5.0 \quad w_D = 1 / (1 - 0.2) = 1.25

Rare users count most: B took treatment despite 0.2, and C skipped it despite 0.8. Weighted treated mean is (10×1.25+14×5.0)/(1.25+5.0)=82.5/6.25=13.2(10 \times 1.25 + 14 \times 5.0) / (1.25 + 5.0) = 82.5 / 6.25 = 13.2. Weighted control mean is (6×5.0+8×1.25)/6.25=40.0/6.25=6.4(6 \times 5.0 + 8 \times 1.25) / 6.25 = 40.0 / 6.25 = 6.4. The weighted gap is 13.2−6.4=6.813.2 - 6.4 = 6.8. Same four rows, different answer, because the weights stand in for the trial that never ran. Matching tells the same story pair by pair: A against C gives 10−6=410 - 6 = 4 at 0.8, B against D gives 14−8=614 - 8 = 6 at 0.2.

Code

The point is the check: rebuild the naive gap 5.0 and the weighted gap 6.8 from the four rows above.

treated: list[tuple[float, float]] = [(10, 0.8), (14, 0.2)]control: list[tuple[float, float]] = [(6, 0.8), (8, 0.2)]

def mean(xs: list[float]) -> float:    return sum(xs) / len(xs)

naive_t: float = mean([v for v, _ in treated])naive_c: float = mean([v for v, _ in control])print(f"{naive_t:.1f} {naive_c:.1f} {naive_t - naive_c:.1f}")  # -> 12.0 7.0 5.0
wt: list[float] = [1 / p for _, p in treated]wc: list[float] = [1 / (1 - p) for _, p in control]print(f"{wt[0]:.2f} {wt[1]:.2f} {wc[0]:.2f} {wc[1]:.2f}")  # -> 1.25 5.00 5.00 1.25
adj_t: float = sum(v * w for (v, _), w in zip(treated, wt)) / sum(wt)adj_c: float = sum(v * w for (v, _), w in zip(control, wc)) / sum(wc)print(f"{adj_t:.1f} {adj_c:.1f} {adj_t - adj_c:.1f}")  # -> 13.2 6.4 6.8

The printout matches the hand arithmetic. Move B outcome from 14 to 10 and watch the weighted gap fall from 6.8 to 3.6, which shows how much rides on one rare row.

Watch Out For

A confounder that was never measured

Scores balance age and income perfectly, overlap looks clean, and the estimate still misses by a mile.

Symptom: matched pairs differ on something the table never held, like motivation or night shift work, and the gap moves whenever a new column arrives. Fix: name the adjustment set from a graph before fitting, never from column availability, and say plainly what would break the result. Scores cannot balance what was never recorded. An instrumental variable design or a real trial is the sequel when the missing trait matters.

Wild weights from scores near zero or one

One user scores 0.99 or 0.01 and the weight reads 100. The estimate swings with every rerun and confidence intervals widen past usefulness.

Symptom: a handful of rows own most of the weight sum, and dropping three rows moves the gap by points. Fix: trim scores outside 0.05 to 0.95 or stabilize weights by their mean, then report the overlap population and the trim rate. If trimming drops a tenth of treated users, the question changed, so state the new one.

The Quick Version

  • The propensity score e(X)=P(T=1∣X)e(X) = P(T = 1 \mid X) compresses background traits into one treatment probability in [0,1][0, 1].
  • Matching compares treated and control users at the same score, while weighting rebalances the sample with 1/e(X)1 / e(X) and 1/(1−e(X))1 / (1 - e(X)).
  • Our four users at 0.8 and 0.2 give naive means 12.0 and 7.0 for gap 5.0.
  • Weights 1.25 and 5.0 give adjusted means 13.2 and 6.4 for gap 6.8.
  • No overlap means no twin, so drop the region and rename the population.
  • Unmeasured traits and extreme weights break the estimate, so graph the adjustment set and trim near zero or one.