Calibration
A 90% claim should be right nine times in ten. Calibration checks whether it is, and fixes the gap when it is not.
Why Does This Exist?
You can hit 90% accuracy and still not trust a single number the model says. Picture ten cases where the model claims 0.90 and gets six right. The accuracy column looks fine, yet every 0.90 you act on is wrong four times in ten. If a 0.90 cut auto approves refunds, flags patients for review, or ships code, you spend real effort on a number that does not mean what it says.
A quick grounding first. For each input with true label , the model outputs logits , meaning one raw score per class. The predicted class is the argmax, meaning the class with the largest score, and the confidence is the top softmax probability, meaning the share given to that class. Accuracy asks how often equals . Calibration asks whether sits beside a correct prediction nine times in ten. If you have not priced right against wrong yet, start with classification metrics.
This page measures the gap with a reliability diagram and one number, expected calibration error, then applies the simplest fix, temperature scaling. Threshold selection is where you price each mistake and place the cut. Conformal prediction skips single numbers and returns sets with a guarantee. Probability calibration fits richer maps when one dial is not enough.
Think of It Like This
The weather forecaster
You trust a forecaster who says 80% exactly when rain comes four days out of five. Now she announces 90% on ten scattered days and rain comes on six. Her hit rate is decent, but her 90% does not mean 90%. You would carry cover on her 90% days and still get soaked four times in ten.
Each announced figure is the confidence, each rainy day is a correct prediction, and the ten day record is one bin. Recalibrating her means adjusting her dial so future 90% calls rain nine times in ten.
Where it stops: you cannot fix a person with one division, and honesty is her whole job. A network trained to minimize log loss is rewarded for sharper numbers, so its dial drifts for a different reason, and one tuned divisor can straighten the model in a way no pep talk straightens a person.
How It Actually Works
Perfect calibration has one equation. Write for every in . In words: among all predictions stated with confidence , the fraction correct equals . A reliability diagram checks this bin by bin. Group predictions by stated confidence, plot mean confidence on the horizontal axis against fraction correct on the vertical axis, and read distance from the diagonal. Points below the diagonal read high and are overconfident. Points above it read low and are underconfident.
The single number version is expected calibration error:
Here is the bin count, is bin , is its size, is the total, is the fraction correct in the bin, and is the mean confidence there. An ECE of 0.00 is perfect and 0.20 means stated confidence misses by twenty points on average.
What the diagram is showing
The horizontal axis is what the model promised and the vertical axis is what happened. The dashed diagonal is perfect: promise , deliver . Our two model dots sit below it at and , so both bins run hot and the top bin runs hotter. Each vertical dashed segment is one bin gap, 0.30 and 0.10. The headline number 0.20 is the weighted mean of those two gaps.
Worked example: ten predictions with ECE 0.20
Take ten held out predictions in two bins:
- High bin, stated range above 0.80 up to 1.00, five cases with confidences and outcomes , meaning three right out of five.
- Mid bin, stated range 0.60 to 0.80, five cases with confidences and outcomes , meaning three right out of five.
Now the arithmetic. The high sum is , so mean confidence is . Accuracy is . The gap is with weight , so it pays . The mid sum is , so mean confidence is . Accuracy is again . The gap is with weight , so it pays . Add them:
That 0.20 is the diagram headline. Same ten numbers appear in the picture, in the code below, and in this section, so all three check each other.
The one dial fix: temperature scaling
Temperature scaling divides every logit by one tuned number before the softmax:
Here holds the raw class scores, is the temperature, and is the rescaled probability for class . Setting changes nothing. A above 1 softens peaks and pulls 0.90 down toward 0.60. A below 1 sharpens them. The argmax never moves, so accuracy and ranking stay put while confidence shifts, which is why this fix cannot repair a bad classifier.
Tune on held out validation by minimizing log loss, then report ECE on a third split neither training nor tuning has touched. If some classes run hot while others run cold, one dial underfits and you want the classwise maps in probability calibration. When inputs drift or a wrong 0.90 carries real cost, pair any rescaling with conformal prediction and with the uncertainty split in aleatoric vs epistemic uncertainty.
Code
The point is the check: rebuild the two bins from the worked example and confirm the hand computation of 0.20.
import numpy as np
conf: np.ndarray = np.array([0.95, 0.95, 0.90, 0.85, 0.85, 0.75, 0.72, 0.70, 0.68, 0.65])correct: np.ndarray = np.array([1, 1, 1, 0, 0, 1, 1, 1, 0, 0])
high: np.ndarray = conf > 0.80high_conf: float = float(conf[high].mean())high_acc: float = float(correct[high].mean())low_conf: float = float(conf[~high].mean())low_acc: float = float(correct[~high].mean())ece: float = float(0.5 * abs(high_conf - high_acc) + 0.5 * abs(low_conf - low_acc))
print(f"{high_conf:.2f} {high_acc:.2f}") # -> 0.90 0.60print(f"{low_conf:.2f} {low_acc:.2f}") # -> 0.70 0.60print(f"{ece:.2f}") # -> 0.20The printout matches the hand computation above: high bin 0.90 confidence at 0.60 accuracy, mid bin 0.70 confidence at 0.60 accuracy, ECE 0.20.
Watch Out For
Trusting one ECE number across bin choices
ECE moves when the bins move. The same ten predictions can read near 0.15 with wide bins and near 0.25 with narrow ones, and a single mean can hide one hot class cancelling one cold class.
Symptom: model A beats model B under ten bins and loses under fifteen, or ECE reads 0.02 while the top bucket still misses by twenty points. Fix: freeze the binning before you compare anything, plot the reliability diagram beside the number, and check classwise gaps plus the Brier score before you ship the verdict.
Fitting the fix on training predictions
A calibrator fit on the same rows the model trained on learns optimism it will never meet again. The curve snaps onto the diagonal in the notebook and live 90% calls land near 60%.
Symptom: validation ECE looks fixed at 0.03 while the first live week reads 0.18, and the 0.90 auto approve cut lets through far more misses than priced. Fix: fit the temperature on held out validation, measure on a third split, and recheck after any shift. When a miss carries real cost, use conformal prediction sets instead of a single rescaled number.
The Quick Version
- Calibration means a stated 0.90 is right nine times in ten, and accuracy alone cannot confirm that.
- Our ten predictions split into bins at 0.90 and 0.70 mean confidence with 0.60 accuracy in each, so the gaps are 0.30 and 0.10.
- Expected calibration error is the size weighted mean of those gaps, here .
- Temperature scaling divides all logits by one validation tuned and leaves the predicted class unchanged.
- A single ECE hides binning choices and classwise gaps, and any fix fit on training rows fails live, so plot the diagram and fit on held out data.