Skip to content
AI360XpertBeta

Calibration

A 90% claim should be right nine times in ten. Calibration checks whether it is, and fixes the gap when it is not.

Ten predictions at 0.90 and 0.70 confidence land at 0.60 accuracy, giving ECE 0.20
Ten predictions at 0.90 and 0.70 confidence land at 0.60 accuracy, giving ECE 0.20

Why Does This Exist?

You can hit 90% accuracy and still not trust a single number the model says. Picture ten cases where the model claims 0.90 and gets six right. The accuracy column looks fine, yet every 0.90 you act on is wrong four times in ten. If a 0.90 cut auto approves refunds, flags patients for review, or ships code, you spend real effort on a number that does not mean what it says.

A quick grounding first. For each input xx with true label yy, the model outputs logits zz, meaning one raw score per class. The predicted class y^\hat{y} is the argmax, meaning the class with the largest score, and the confidence p^\hat{p} is the top softmax probability, meaning the share given to that class. Accuracy asks how often y^\hat{y} equals yy. Calibration asks whether p^=0.90\hat{p} = 0.90 sits beside a correct prediction nine times in ten. If you have not priced right against wrong yet, start with classification metrics.

This page measures the gap with a reliability diagram and one number, expected calibration error, then applies the simplest fix, temperature scaling. Threshold selection is where you price each mistake and place the cut. Conformal prediction skips single numbers and returns sets with a guarantee. Probability calibration fits richer maps when one dial is not enough.

Think of It Like This

The weather forecaster

You trust a forecaster who says 80% exactly when rain comes four days out of five. Now she announces 90% on ten scattered days and rain comes on six. Her hit rate is decent, but her 90% does not mean 90%. You would carry cover on her 90% days and still get soaked four times in ten.

Each announced figure is the confidence, each rainy day is a correct prediction, and the ten day record is one bin. Recalibrating her means adjusting her dial so future 90% calls rain nine times in ten.

Where it stops: you cannot fix a person with one division, and honesty is her whole job. A network trained to minimize log loss is rewarded for sharper numbers, so its dial drifts for a different reason, and one tuned divisor can straighten the model in a way no pep talk straightens a person.

How It Actually Works

Perfect calibration has one equation. Write P(y=y^∣p^=p)=pP(y = \hat{y} \mid \hat{p} = p) = p for every pp in [0,1][0, 1]. In words: among all predictions stated with confidence pp, the fraction correct equals pp. A reliability diagram checks this bin by bin. Group predictions by stated confidence, plot mean confidence on the horizontal axis against fraction correct on the vertical axis, and read distance from the diagonal. Points below the diagonal read high and are overconfident. Points above it read low and are underconfident.

The single number version is expected calibration error:

ECE=∑m=1M∣Bm∣n ∣acc(Bm)−conf(Bm)∣\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \, \lvert \mathrm{acc}(B_m) - \mathrm{conf}(B_m) \rvert

Here MM is the bin count, BmB_m is bin mm, ∣Bm∣|B_m| is its size, nn is the total, acc\mathrm{acc} is the fraction correct in the bin, and conf\mathrm{conf} is the mean confidence there. An ECE of 0.00 is perfect and 0.20 means stated confidence misses by twenty points on average.

What the diagram is showing

The horizontal axis is what the model promised and the vertical axis is what happened. The dashed diagonal is perfect: promise pp, deliver pp. Our two model dots sit below it at (0.90,0.60)(0.90, 0.60) and (0.70,0.60)(0.70, 0.60), so both bins run hot and the top bin runs hotter. Each vertical dashed segment is one bin gap, 0.30 and 0.10. The headline number 0.20 is the weighted mean of those two gaps.

Worked example: ten predictions with ECE 0.20

Take ten held out predictions in two bins:

  • High bin, stated range above 0.80 up to 1.00, five cases with confidences [0.95,0.95,0.90,0.85,0.85][0.95, 0.95, 0.90, 0.85, 0.85] and outcomes [1,1,1,0,0][1, 1, 1, 0, 0], meaning three right out of five.
  • Mid bin, stated range 0.60 to 0.80, five cases with confidences [0.75,0.72,0.70,0.68,0.65][0.75, 0.72, 0.70, 0.68, 0.65] and outcomes [1,1,1,0,0][1, 1, 1, 0, 0], meaning three right out of five.

Now the arithmetic. The high sum is 0.95+0.95+0.90+0.85+0.85=4.500.95 + 0.95 + 0.90 + 0.85 + 0.85 = 4.50, so mean confidence is 4.50/5=0.904.50 / 5 = 0.90. Accuracy is 3/5=0.603 / 5 = 0.60. The gap is ∣0.90−0.60∣=0.30\lvert 0.90 - 0.60 \rvert = 0.30 with weight 5/105 / 10, so it pays 0.5×0.30=0.150.5 \times 0.30 = 0.15. The mid sum is 0.75+0.72+0.70+0.68+0.65=3.500.75 + 0.72 + 0.70 + 0.68 + 0.65 = 3.50, so mean confidence is 3.50/5=0.703.50 / 5 = 0.70. Accuracy is again 3/5=0.603 / 5 = 0.60. The gap is ∣0.70−0.60∣=0.10\lvert 0.70 - 0.60 \rvert = 0.10 with weight 5/105 / 10, so it pays 0.5×0.10=0.050.5 \times 0.10 = 0.05. Add them:

ECE=0.15+0.05=0.20\mathrm{ECE} = 0.15 + 0.05 = 0.20

That 0.20 is the diagram headline. Same ten numbers appear in the picture, in the code below, and in this section, so all three check each other.

The one dial fix: temperature scaling

Temperature scaling divides every logit by one tuned number T>0T > 0 before the softmax:

pi=softmax(z/T)ip_i = \mathrm{softmax}(z / T)_i

Here zz holds the raw class scores, TT is the temperature, and pip_i is the rescaled probability for class ii. Setting T=1T = 1 changes nothing. A TT above 1 softens peaks and pulls 0.90 down toward 0.60. A TT below 1 sharpens them. The argmax never moves, so accuracy and ranking stay put while confidence shifts, which is why this fix cannot repair a bad classifier.

Tune TT on held out validation by minimizing log loss, then report ECE on a third split neither training nor tuning has touched. If some classes run hot while others run cold, one dial underfits and you want the classwise maps in probability calibration. When inputs drift or a wrong 0.90 carries real cost, pair any rescaling with conformal prediction and with the uncertainty split in aleatoric vs epistemic uncertainty.

Code

The point is the check: rebuild the two bins from the worked example and confirm the hand computation of 0.20.

import numpy as np
conf: np.ndarray = np.array([0.95, 0.95, 0.90, 0.85, 0.85, 0.75, 0.72, 0.70, 0.68, 0.65])correct: np.ndarray = np.array([1, 1, 1, 0, 0, 1, 1, 1, 0, 0])
high: np.ndarray = conf > 0.80high_conf: float = float(conf[high].mean())high_acc: float = float(correct[high].mean())low_conf: float = float(conf[~high].mean())low_acc: float = float(correct[~high].mean())ece: float = float(0.5 * abs(high_conf - high_acc) + 0.5 * abs(low_conf - low_acc))
print(f"{high_conf:.2f} {high_acc:.2f}")  # -> 0.90 0.60print(f"{low_conf:.2f} {low_acc:.2f}")  # -> 0.70 0.60print(f"{ece:.2f}")  # -> 0.20

The printout matches the hand computation above: high bin 0.90 confidence at 0.60 accuracy, mid bin 0.70 confidence at 0.60 accuracy, ECE 0.20.

Watch Out For

Trusting one ECE number across bin choices

ECE moves when the bins move. The same ten predictions can read near 0.15 with wide bins and near 0.25 with narrow ones, and a single mean can hide one hot class cancelling one cold class.

Symptom: model A beats model B under ten bins and loses under fifteen, or ECE reads 0.02 while the top bucket still misses by twenty points. Fix: freeze the binning before you compare anything, plot the reliability diagram beside the number, and check classwise gaps plus the Brier score before you ship the verdict.

Fitting the fix on training predictions

A calibrator fit on the same rows the model trained on learns optimism it will never meet again. The curve snaps onto the diagonal in the notebook and live 90% calls land near 60%.

Symptom: validation ECE looks fixed at 0.03 while the first live week reads 0.18, and the 0.90 auto approve cut lets through far more misses than priced. Fix: fit the temperature on held out validation, measure on a third split, and recheck after any shift. When a miss carries real cost, use conformal prediction sets instead of a single rescaled number.

The Quick Version

  • Calibration means a stated 0.90 is right nine times in ten, and accuracy alone cannot confirm that.
  • Our ten predictions split into bins at 0.90 and 0.70 mean confidence with 0.60 accuracy in each, so the gaps are 0.30 and 0.10.
  • Expected calibration error is the size weighted mean of those gaps, here 0.5×0.30+0.5×0.10=0.200.5 \times 0.30 + 0.5 \times 0.10 = 0.20.
  • Temperature scaling divides all logits by one validation tuned TT and leaves the predicted class unchanged.
  • A single ECE hides binning choices and classwise gaps, and any fix fit on training rows fails live, so plot the diagram and fit on held out data.