Precision-Recall Curve

Neither axis counts the easy negatives you got right, so a mountain of obvious no cases cannot flatter the picture.

With 900 leavers recall holds at 0.73 while precision falls from 0.50 to 0.23 as stayers grow, and chance sits at 0.09 not one half

Why Does This Exist?

Ten thousand subscribers hold 900 future cancellations. A model scores churn risk and the top slice gets a retention offer. Two numbers decide the budget. Of the offers sent, how many reached real leavers. Of the 900 leavers, how many got an offer.

Grow the base to thirty thousand with the same 900 leavers. A ROC and AUC view barely moves because its false positive rate divides by tens of thousands of stayers. Precision falls from 0.50 to 0.23 because its denominator counts only offers sent. One view says nothing changed. The other says the economics broke.

This page covers the view that noticed. The two by two counts behind it live in confusion matrix. The single number summaries live in classification metrics. Picking one operating cut from costs lives in threshold selection.

Think of It Like This

Panning two bends of the same river

You pan for gold with one fixed technique. Two numbers describe the day: of what stayed in the pan, how much was gold, and of the gold the river carried past, how much you kept. Move to a richer bend with the same skill and the pan suddenly holds mostly gold. Move to a poor stretch and it holds grit. Skill never changed, the river did.

Neither number weighs the tonnes of plain gravel you correctly let wash away. Thousands of correct rejections an hour appear nowhere. That missing pile is the point: the score describes what you kept against what mattered, never the gravel you ignored.

Where it stops: a pan cannot show how much gold you never touched upstream. This curve cannot score thresholds you never tried, and it never prices the offer itself.

How It Actually Works

Call $TP$ true positives, meaning leavers flagged. Call $FP$ false positives, meaning stayers flagged. Call $FN$ false negatives, meaning leavers missed. Precision means flags that were right. Recall means leavers reached:

\text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}

True negatives, meaning stayers correctly left alone, appear in neither denominator. That absence is the whole difference from ROC.

Worked example: same leavers, more stayers

Fix the model cut so it catches $TP = 655$ of 900 leavers, missing $FN = 245$ . Recall is:

655 / 900 \approx 0.728 \approx 0.73

On the small base the cut also flags $FP = 650$ stayers. Precision is:

655 / (655 + 650) = 655 / 1305 \approx 0.502 \approx 0.50

Now hold the cut and add twenty thousand extra stayers the model never saw. The same cut adds $FP = 1530$ more false flags, so total $FP = 2180$ . Recall cannot move because its denominator is still the same 900 leavers. Precision becomes:

655 / (655 + 2180) = 655 / 2835 \approx 0.231 \approx 0.23

Recall 0.73 in both worlds, precision 0.50 then 0.23. The false positive rate moves only half a point because its denominator counts every stayer correctly ignored. That is the diagram: one horizontal move in ROC space, one steep fall here.

Why the baseline is the positive rate

Flag at random and the share of flags that hit equals the share of positives in the base. On 900 of 10,000 the chance line sits at $900/10000 = 0.09$ , not 0.50. The standard summary is average precision, meaning the recall weighted mean of precision across cuts. An average precision of 0.40 beats chance fourfold on a 9 percent base and barely beats chance on a 45 percent base, so the number means nothing without its prevalence beside it.

The sawtooth at the strict end

Walk the cut down one case at a time. Each step either hits, nudging precision up, or misses, dragging it down, while recall only climbs. At high cuts the flagged set holds a handful of cases, so one flip moves precision by more than a tenth. The teeth are a small denominator artefact, not model instability. Average precision sums step size in recall times precision at that step, which respects that structure. A trapezoid in this space joins points with straight lines no classifier can realize, so it reads optimistic.

Code

Rebuild the two worlds from the counts above with no sampling and no libraries.

1tp: int = 6552fn: int = 2453fp_small: int = 6504fp_extra: int = 15305
6recall: float = tp / (tp + fn)7prec_small: float = tp / (tp + fp_small)8prec_big: float = tp / (tp + fp_small + fp_extra)9
10print(f"{recall:.2f} {prec_small:.2f}")  # -> 0.73 0.5011print(f"{recall:.2f} {prec_big:.2f}")  # -> 0.73 0.2312print(f"{900/10000:.2f}")  # -> 0.09

Recall prints 0.73 twice while precision prints 0.50 then 0.23, matching the hand division. Chance prints 0.09, the positive rate on the small base.

Watch Out For

Comparing average precision across different positive rates

Symptom: average precision drops from 0.44 to 0.31 with no model change, and a week goes into feature diffs, while churn itself fell from 9 percent to 6 percent. Fix: always quote prevalence beside average precision and compare lift over chance, meaning average precision divided by the positive rate. Never benchmark against a published number from another base.

Smoothing the strict end teeth

Symptom: the spiky corner reads as instability on confident predictions and someone proposes smoothing the curve before plotting. Fix: leave the teeth alone. Precision there divides by fewer than ten flags, so coarse jumps are expected. Report precision at a fixed flag count instead, such as precision among the top 100 scores, which pins the denominator.

The Quick Version

Precision is flags that were right over flags sent, recall is leavers reached over all 900 leavers.
Neither denominator counts stayers correctly ignored, so easy negatives cannot flatter the view.
With 655 hits and 245 misses recall is 655 divided by 900, about 0.73, in both worlds.
Precision is 655 divided by 1305, about 0.50, on the small base and 655 divided by 2835, about 0.23, after stayers grow.
Chance sits at the positive rate, 0.09 on the small base, never at one half.
Average precision needs its prevalence beside it, and strict end teeth reflect tiny denominators.