Propensity Score Methods

Why Does This Exist?

Randomized A/B testing is the gold standard for finding causal effects, but it isn't always possible. You cannot randomly force half your users to smoke cigarettes to see if it causes cancer. You cannot randomly force half your customers to churn to see how it affects your revenue. You are stuck with observational data.

In observational data, the treatment and control groups look totally different. The people who chose to use your new ML feature might be younger, wealthier, and more tech-savvy than the people who didn't. If you just compare their average outcomes, you suffer from massive confounding bias.

Propensity Score Methods exist to artificially reconstruct a randomized experiment out of biased observational data. They do this by finding control users who look exactly like the treated users, and throwing away (or down-weighting) everyone else.

Think of It Like This

How It Actually Works

Finding exact twins in a dataset with 50 features is impossible due to the curse of dimensionality. The Propensity Score Theorem (Rosenbaum and Rubin, 1983) solves this by proving you don't need to match on all 50 features. You only need to match on the probability of receiving the treatment.

1. Calculate the Propensity Score

We train a standard supervised machine learning model (often Logistic Regression or Gradient Boosting).

• Features (X): All the confounding variables (age, income, past behavior).
• Target (Y): Did they receive the treatment? (1 or 0).

The output of this model is the Propensity Score $e(X) = P(T=1 | X)$. It is a single number between 0 and 1 that represents how likely this user was to get the treatment, given their traits.

2. Check for Overlap (Common Support)

If a treated user has a score of 0.99, but the highest control user has a score of 0.40, we cannot match them. They are too fundamentally different. We must ensure there is Overlap (Common Support) in the distributions of propensity scores between the two groups. Users at the extreme edges who have no match must be dropped.

3. Estimate the Effect

Once we have the scores, we can estimate the Average Treatment Effect (ATE) in a few ways:

• Propensity Score Matching (PSM): For every treated user, find the control user with the closest propensity score. Calculate the difference in their outcomes.
• Inverse Probability Weighting (IPW): Create a synthetic population where everyone is treated equally by weighting the data. A treated user with a low propensity score of 0.1 is rare, so they get a massive weight ($1 / 0.1 = 10$). A control user who looked like they should have been treated (score 0.9) also gets a massive weight ($1 / (1 - 0.9) = 10$).

Show Me the Code

Here is a simplified example of Inverse Probability Weighting (IPW) using Logistic Regression to calculate the propensity scores.

1import numpy as np2import pandas as pd3from sklearn.linear_model import LogisticRegression4import statsmodels.api as sm5
6np.random.seed(42)7N = 100008
9# 1. Generate biased observational data10age = np.random.normal(40, 10, N)11income = np.random.normal(60000, 20000, N)12
13# Treatment assignment is highly biased by age and income (Confounders)14logit_p = -3 + 0.05 * (age - 40) + 0.00005 * (income - 60000)15prob_treatment = 1 / (1 + np.exp(-logit_p))16treatment = np.random.binomial(1, prob_treatment)17
18# True Causal Effect is +50019outcome = 1000 + 500 * treatment + 10 * age + 0.05 * income + np.random.normal(0, 100, N)20
21df = pd.DataFrame({'age': age, 'income': income, 'treatment': treatment, 'outcome': outcome})22
23# Naive comparison (Biased)24naive_ate = df[df['treatment']==1]['outcome'].mean() - df[df['treatment']==0]['outcome'].mean()25print(f"Naive ATE: {naive_ate:.2f}") 26# -> ~ +1100 (Massively overestimates the true effect of +500)27
28# 2. Calculate Propensity Scores29X = df[['age', 'income']]30ps_model = LogisticRegression().fit(X, df['treatment'])31df['propensity_score'] = ps_model.predict_proba(X)[:, 1]32
33# 3. Calculate IPW Weights34# Weight for treated: 1 / p35# Weight for control: 1 / (1 - p)36df['weight'] = np.where(df['treatment'] == 1, 37                        1 / df['propensity_score'], 38                        1 / (1 - df['propensity_score']))39
40# 4. Estimate Causal Effect using Weighted Least Squares (WLS)41X_ipw = sm.add_constant(df['treatment'])42ipw_model = sm.WLS(df['outcome'], X_ipw, weights=df['weight']).fit()43
44print(f"IPW ATE (Causal): {ipw_model.params['treatment']:.2f}")45# -> ~ +500 (Recovers the true causal effect!)

Watch Out For

The Quick Version

• Propensity Score: The probability that a user receives the treatment, based solely on their observed characteristics.
• It compresses a high-dimensional set of confounding variables down to a single number between 0 and 1.
• You can use this score to Match treated and control users who are statistically identical, or use Inverse Probability Weighting (IPW) to balance the dataset.
• It only works if you have actually measured all the important confounders, and if there is overlap (common support) between the two groups.

What to Read Next

• difference-in-differences — Another observational causal method that can handle unmeasured confounders, as long as they are stable over time.
• instrumental-variables — The ultimate fallback when you have massive, unmeasured confounding and no overlap.
• confounding-and-colliders — Review which variables you are actually allowed to put into the propensity score model.