Probability Calibration

Ranking tells you which van breaks first. Calibration tells you whether 0.85 really means 85 in 100.

Twelve scores at mean 0.85 and 0.65 land at 0.50 accuracy, giving ECE 0.25

Why Does This Exist?

A parts team orders batteries by adding up model scores across an 8,000 van fleet. The ranking is perfect: the vans scored highest really do fail most. Yet the sum says 6,800 expected failures and 4,250 happen. The order runs 60 percent high because every score reads high, not because the order is wrong.

That split is the whole page. Ranking asks whether failures cluster at the top of the sorted list, which is what ROC and AUC and most of classification metrics measure. Calibration asks whether 0.85 means 85 in 100. Money, staffing, and thresholds multiply the number, so the second question decides the budget while the first only decides the order.

Two prerequisites, briefly. A classifier outputs a score $s$ in $[0, 1]$ for each input $x$ , and the predicted label is 1 when $s$ passes a threshold. Ground truth is the hand checked outcome, 1 for failed and 0 for healthy, without which no rate below can be computed.

This page covers the richer fix: bucketed checks plus Platt and isotonic maps. The single number gap with one dial fix lives in calibration. Placing the cut from costs lives in threshold selection. Returning sets with a guarantee instead of single numbers lives in conformal prediction.

Think of It Like This

A fuel gauge that ranks tanks correctly

Two vans sit in the yard with one suspect gauge. You read both: the van showing more diesel really has more diesel, every time. As a ranking tool it is flawless.

Now you plan a 60 mile route on a quarter tank reading. The van stops at 32 miles. The gauge never lied about which tank was fuller. It lied about how much was inside, and the moment you multiplied the reading the ranking stopped mattering.

The fix is physical. Pour 10 measured litres, note the needle, repeat at 20 and 30, then plot needle against litres poured. A straight line means honesty. Points sagging under the diagonal mean it reads high, and the plot becomes a lookup table that undoes the lie. That plot is a reliability diagram.

Where it stops: a gauge has one needle and one tank shape. A model has thousands of inputs and its error changes with the input, so one lookup table per score range replaces one bend in a needle, and the table must be refit when the fleet changes.

How It Actually Works

The reliability diagram

One prediction cannot be checked. A van either fails or it does not, so 0.85 on one van is unfalsifiable. Calibration is a property of groups, which is why the check needs buckets.

Take held out scored cases the model never trained on. Sort by score, cut into equal count bins, and per bin plot mean score on the horizontal axis against fraction positive on the vertical axis. Perfect calibration lands on the diagonal. Points below it read high. Points above it read low.

Worked example: twelve vans, ECE 0.25

Take twelve held out vans in two bins of six. Write $s$ for the score and $y$ for the outcome, 1 for failed and 0 for healthy.

High bin scores are $[0.90, 0.90, 0.85, 0.85, 0.80, 0.80]$ with outcomes $[1, 1, 1, 0, 0, 0]$ , meaning three failed out of six. Add them:

0.90 + 0.90 + 0.85 + 0.85 + 0.80 + 0.80 = 5.10

Divide by six:

\bar{s}_{high} = 5.10 / 6 = 0.85

Accuracy is $3 / 6 = 0.50$ . The gap is $\lvert 0.85 - 0.50 \rvert = 0.35$ with weight $6 / 12$ , so it pays $0.5 \times 0.35 = 0.175$ .

Mid bin scores are $[0.70, 0.70, 0.65, 0.65, 0.60, 0.60]$ with outcomes $[1, 1, 1, 0, 0, 0]$ . The sum is $0.70 + 0.70 + 0.65 + 0.65 + 0.60 + 0.60 = 3.90$ , so mean is $3.90 / 6 = 0.65$ . Accuracy is again $3 / 6 = 0.50$ . The gap is $\lvert 0.65 - 0.50 \rvert = 0.15$ with weight $6 / 12$ , so it pays $0.5 \times 0.15 = 0.075$ . Add them:

\mathrm{ECE} = 0.175 + 0.075 = 0.25

Here $\mathrm{ECE}$ is expected calibration error, meaning the size weighted mean gap between stated and observed rates. The diagram shows the same two dots at $(0.85, 0.50)$ and $(0.65, 0.50)$ below the diagonal, with vertical gaps 0.35 and 0.15 and headline 0.25.

Platt scaling and isotonic regression

Both fixes fit a one input map from raw score to probability on data the model never saw.

Platt scaling fits a logistic regression on the score, two parameters. A few hundred held out rows suffice. It only undoes S shaped distortion, which fits margin models and boosted trees well.

Isotonic regression fits the best non decreasing step function and assumes nothing beyond monotone order. It straightens bends Platt cannot reach, and it overfits fast. Under about a thousand held out rows, use Platt.

Score the result with the Brier score, the mean squared gap between probability $p$ and outcome $y$ in $\{0, 1\}$ . It is a proper scoring rule, meaning it bottoms out only when you report what you believe. The diagram is the diagnosis and Brier is the verdict. Logistic regression trained by maximum likelihood starts near calibrated because its loss rewards honest frequencies, while naive Bayes double counts correlated evidence and piles scores at the ends.

Code

The point is the check: rebuild both bins from the worked example and confirm 0.85 against 0.50, 0.65 against 0.50, and ECE 0.25.

1def mean(xs: list[float]) -> float:2    """Plain mean, so the bin arithmetic stays visible."""3    return sum(xs) / len(xs)4
5
6high: list[float] = [0.90, 0.90, 0.85, 0.85, 0.80, 0.80]7mid: list[float] = [0.70, 0.70, 0.65, 0.65, 0.60, 0.60]8high_hit: list[float] = [1, 1, 1, 0, 0, 0]9mid_hit: list[float] = [1, 1, 1, 0, 0, 0]10
11high_conf: float = mean(high)12high_acc: float = mean(high_hit)13mid_conf: float = mean(mid)14mid_acc: float = mean(mid_hit)15ece: float = 0.5 * abs(high_conf - high_acc) + 0.5 * abs(mid_conf - mid_acc)16
17print(f"{high_conf:.2f} {high_acc:.2f}")  # -> 0.85 0.5018print(f"{mid_conf:.2f} {mid_acc:.2f}")  # -> 0.65 0.5019print(f"{ece:.2f}")  # -> 0.25

The printout matches the hand computation: high bin promises 0.85 and delivers 0.50, mid bin promises 0.65 and delivers 0.50, weighted gap 0.25. Change one high outcome from 0 to 1 and watch the high gap fall from 0.35 to 0.18.

Watch Out For

Fitting the map on training scores

The curve snaps onto the diagonal in the notebook and Brier improves, then the first live week reads high by the same amount as before.

Symptom: validation ECE near 0.03 while live ECE reads near 0.20, with the 0.85 bucket missing by thirty points. Fix: fit Platt or isotonic only on held out scores or out of fold predictions, then measure ECE and Brier on a third split neither the model nor the map has touched, as in train validation and test splits.

Resampling for rare failures, then trusting the numbers

Failures run near 1 in 25, so the team downsamples healthy vans. Recall improves and AUC barely moves.

Symptom: ranking holds while every budget calc runs high, often near 0.30 predicting 0.04 in the wild. Fix: correct the intercept back to the true base rate after any reweighting, or refit the calibration map on untouched prevalence. With severe skew, read imbalanced data before trusting a resampled score.

The Quick Version

Ranking and calibration are separate: 0.85 must mean 85 failures in 100, and AUC cannot confirm that.
The reliability diagram buckets scores and plots mean score against observed rate, reading distance from the diagonal.
Our twelve vans split into 0.85 and 0.65 mean confidence with 0.50 accuracy in each, so gaps are 0.35 and 0.15.
Expected calibration error is the size weighted mean of those gaps, here $0.5 \times 0.35 + 0.5 \times 0.15 = 0.25$ .
Platt scaling fits two parameters and assumes an S shape, while isotonic fits any monotone steps and needs about a thousand rows.
Any map fit on training scores fails live, so fit on held out data and confirm with Brier on a third split.