Disparate Impact in AI Models

A hiring rule can ignore gender on paper and still reject women at far higher rates. Disparate impact is that gap in outcomes, measured by selection rates rather than by intent.

One neutral threshold passes 80 percent of one group but only 30 percent of another, giving a disparate impact ratio of 0.375.

Why Does This Exist?

You remove gender from a hiring model and the gap barely moves. Women still get offers at a third the rate of men. That is the moment this idea earns its keep.

Direct discrimination is easy to name: the model reads a protected attribute and penalizes it. That is disparate treatment, and dropping the column stops it. But a model that never sees gender can still sort by it through stand-ins. Zip code, school, job title history, and purchase patterns each carry demographic signal. A single cutoff on a score built from those stand-ins then lands unevenly.

This page covers outcome gaps from neutral rules, how to measure them with selection rates, and where the stand-ins enter. For the broader bias story, start with algorithmic bias. For which fairness definition you are actually using, read individual vs group fairness next. For how to catch this before launch, model auditing is the sequel.

Think of It Like This

One height line for firefighter applicants

A fire department sets one rule: every applicant must be at least 5 feet 10 inches tall. The form never asks for gender. The line applies to all.

Because adult height distributions differ by sex, that single line cuts far deeper into female applicants than male applicants. Same ruler, very different pass rates. The rule is blind. The outcome is not.

So it is with a proxy score. The model draws one line on a score that already leans by group, and the pass rates split. The line looks neutral. The split is the impact.

Where it stops: height is visible and one-dimensional, while model scores blend dozens of stand-ins. You cannot see the split by eyeballing a cutoff. You have to compute the rates per group.

How It Actually Works

Write $Y = 1$ for the favorable decision, such as an offer or a loan approval. Write $A$ for group membership, with $A = 0$ for the higher-selected group and $A = 1$ for the lower-selected group. The selection rate for a group is the share who get $Y = 1$ :

P(Y = 1 \mid A = 0) \quad \text{and} \quad P(Y = 1 \mid A = 1)

In words: among group $A = 0$ , what fraction is selected, and among group $A = 1$ , what fraction is selected. The disparate impact ratio, called DIR, divides the lower by the higher:

\text{DIR} = \frac{P(Y = 1 \mid A = 1)}{P(Y = 1 \mid A = 0)}

The US four-fifths rule flags $\text{DIR} < 0.8$ as evidence of adverse impact. It is a screening rule, not proof of unlawful discrimination, but teams use it as the tripwire.

Where the gap comes from

A proxy variable is an input that correlates with $A$ without naming it. The model learns $P(Y \mid X)$ from historical labels, where $X$ includes the proxy. If past decisions favored one group, $X$ already encodes that history, and the learned cutoff reproduces it. Deleting $A$ from $X$ leaves the proxy path intact, which is why blindness alone does not fix the rates.

Worked example with forty and fifty applicants

Take 50 applicants in group $A = 0$ and 40 in group $A = 1$ . The model approves 40 in group $A = 0$ and 12 in group $A = 1$ .

Selection rates:

Group $A = 0$ : $40 / 50 = 0.80$ , or 80 percent.
Group $A = 1$ : $12 / 40 = 0.30$ , or 30 percent.

Now the ratio:

\text{DIR} = \frac{0.30}{0.80} = \frac{30}{80} = \frac{3}{8} = 0.375

Since $0.375 < 0.8$ , this fails the four-fifths screen by a wide margin. The diagram shows the same two bars, 80 percent against 30 percent, with the 0.375 ratio labeled. Note what the ratio does not say: it does not name the cause. It tells you to go find the proxy and the history behind it.

Watch Out For

Deleting the sensitive column and declaring victory

Symptom: you drop gender, retrain, and the DIR moves from 0.36 to 0.38. The launch note says the model is now blind and therefore fair.

Fix: measure rates per group before and after, and list the three inputs most correlated with group membership. If zip code or school predicts $A$ well, blindness changed nothing. Track DIR on held-out decisions, not on training labels, and treat any DIR below 0.8 as a stop signal.

Fixing rates with per-group cutoffs and no legal review

Symptom: you set a lower cutoff for the lower-selected group until DIR reaches 0.9. Rates look equal, but the system now reads $A$ at decision time to pick the cutoff.

Fix: pause and get a legal review before you ship separate thresholds. In some hiring and lending regimes, explicit use of $A$ at decision time carries its own liability. Prefer fixes that do not need $A$ at serving time, such as reweighting training rows, removing the worst proxy, or gathering better labels, and document which path you chose.

The Quick Version

Disparate impact is an outcome gap from a neutral-looking rule, judged by rates rather than by intent.
Disparate treatment needs explicit use of a protected attribute, while disparate impact does not.
Proxy variables let a blind model reconstruct group membership and reproduce history.
The four-fifths rule flags a disparate impact ratio below 0.8, as in 0.30 divided by 0.80 equals 0.375.
Removing the sensitive column alone rarely moves the ratio, because the proxies remain.
Per-group cutoffs can equalize rates but may need the protected attribute at decision time, so review them first.