LLM Confidence & Calibration
If you say 80% when you mean it, people can plan around you. Calibration is whether a model's 80% means 80%.
Why Does This Exist?
You want to act on a score. Auto close the ticket when confidence tops 90%, pull a human in when it drops under 60%. You'll route staffing, refunds, and escalations off that number, so it has to mean what it says.
Picture 100 answers marked 90%. You expect about 10 mistakes. If the model actually gets 40 wrong, your threshold didn't miss by a little. It understated mistakes by a factor of four. Don't ship a policy like that on tone alone.
Calibration is whether stated confidence matches hit rate. This page covers that match for chat models: how you read it, why chat tuning bends it, and how you straighten it. The general metric with bins and diagrams, no LLM specifics, lives in probability calibration. Why fluent falsehoods sound so sure lives in hallucination mechanisms.
One idea first. A chat model scores each possible next token with raw numbers called logits, then turns them into probabilities with softmax. Bigger logit, bigger share, and the shares sum to 1. If that step feels hazy, read temperature and sampling first, then come back. Everything below assumes scores become probabilities.
Think of It Like This
A weather forecaster who wants to sound sure
You follow a forecaster who says 80% rain. Over 100 such days, it should rain about 80 times. You keep a scorecard, forecast against outcome, and you soon know whether to carry an umbrella when they say 70%.
Then the station starts ranking forecasters by how decisive they sound. Hedging gets booed. Your forecaster learns to say 99% with a straight face on a 60% day. The delivery gets smoother and the scorecard gets worse. Sounding sure paid. Being right didn't.
Every part maps. The forecast percent is stated confidence. Rain frequency is accuracy. Chasing decisiveness is preference tuning. The scorecard is a reliability diagram.
Where it stops: the forecaster feels doubt and hides it. A model feels nothing. It holds token statistics, not nerves. And one rainy day repeats in a way one trivia answer never does, which is why you must group answers before the scorecard means anything.
How It Actually Works
A claim of 70% on one answer is untestable. The answer is right or wrong. Calibration is a claim about groups: across answers where you said about , about fraction should be right. In symbols, . Here means long run fraction, means the answer matches fact, and means the model attached confidence .
Confidence is a group claim
For each answer , write for its stated confidence, a number between 0 and 1 that the model attaches, and for its correctness, which is 1 when the answer is right and 0 when it is wrong. You can't judge one pair. A 0.70 on a single answer is neither true nor false until you see many answers given at 0.70.
That is the whole assumption, and it fails in a specific way. Single answers carry no frequency. If you treat one 95% as a promise, you'll be fooled by design. Group first, then judge.
Bins make it checkable
Sort answers by and cut them into bins. Call bin by the name . Write for how many answers landed in it, for the total number of answers, for the mean of inside it, and for the mean of inside it. In plain words: mean stated confidence versus mean hit rate, per group.
Plot on the x axis and on the y axis. Perfect calibration is the diagonal. Points sagging below it read high. That plot is a reliability diagram.
One number summarizes it. Expected Calibration Error, shortened to ECE, is the weighted mean gap:
Each term says how far this bin sits off the diagonal, weighted by what share of answers it holds. ECE of 0 means the stated numbers match reality. ECE of 0.15 means you are off by 15 points on average.
Worked example: twenty answers and an ECE of 0.15
Take 20 factual answers, already scored, split exactly as drawn above.
High bin: 10 answers stated at 0.90. Six are right. So and . The gap is . Weight is .
Mid bin: 10 answers stated at 0.60. Six are right. So and . The gap is . Weight is .
Now combine:
Check the weights sum to 1: . The diagram shows the same two dots: the 90% dot hangs 0.30 under the diagonal, the 60% dot sits on it. High confidence overpromises. Mid confidence tells the truth.
Why chat tuning bends the numbers
Two confidences drift apart. Verbalized confidence is the words, like "I am 99% sure." Logprob confidence is the math, the probability the model assigned to the tokens it emitted. You'll find the words almost useless for gating. Preference tuning rewards helpful, decisive answers, so hedging gets trained away even where hedging would be honest.
The math bends too. The same tuning pushes token probabilities toward extremes to force decisive generation. Nuanced middles like 40% thin out. High bins fill with answers that should have stayed in the middle, which is exactly the sag you computed above.
Here is the trade off. Decisive tone buys satisfaction and speed. It costs honesty about doubt. The standing assumption is that your questions resemble the tuning distribution. Ask about something sparse, recent, or invented, and that assumption fails silently. Don't use an ungrounded model for cited facts.
Temperature scaling straightens them
Write for the raw logit of candidate token , any size, bigger meaning more plausible here, and for temperature, a positive knob you set. Temperature scaling replaces each probability with:
Here is the exponential function, which turns score gaps into ratios, and the sum over in the denominator forces the values to sum to 1. At nothing changes. At above 1, gaps shrink and the distribution softens. At below 1, gaps widen and it sharpens.
You fit on held out prompts by minimizing log loss, then freeze it. Ranking never changes, because dividing every by the same keeps their order. That is also its limit. When not to use it: when the distortion is not sigmoid shaped, when your prompts drifted from the held out set, or when different topics miscalibrate differently. Then you want binned remapping or sampled agreement instead: ask the same question several times at nonzero temperature and treat disagreement as doubt. It costs several calls per question, so use it where a wrong yes hurts, not everywhere. The judge pattern has the same need, which is why LLM-as-a-Judge scores must be calibrated before you trust their rank order.
Code
This snippet recomputes the worked example directly, so you can check the arithmetic on your own bins.
def bin_gap(conf: float, hits: int, total: int) -> tuple[float, float]: """Return (accuracy, gap) for one bin.""" acc = hits / total return acc, abs(acc - conf)
def ece_from_bins(bins: list[tuple[float, int, int]], total: int) -> float: """Weighted mean gap. Each bin is (confidence, hits, count).""" return sum((count / total) * bin_gap(conf, hits, count)[1] for conf, hits, count in bins)
bins = [(0.90, 6, 10), (0.60, 6, 10)]print(f"{ece_from_bins(bins, 20):.2f}") # -> 0.15The print shows 0.15, matching the hand calculation. Change the counts and the result moves with them, which is the point: ECE is a weighted mean, not a property of one answer.
Watch Out For
Gating on tone instead of measured confidence
Symptom: a fabricated legal citation sails through review because it reads crisp and certain. The model said "I am absolutely sure" with the same polish it gives a true answer, so your 90% auto approve rule fires on words, not math.
Fix: never let verbalized percent drive a decision. Gate on logprob based scores or sampled agreement that you binned on held out prompts, and keep tone out of the threshold entirely.
Fitting the fix on the data you test
Symptom: the notebook shows ECE near zero, then production reads 10 to 15 points high again. The calibrator learned the training distortion, not the live one, or each bin held five answers and the diagram was noise.
Fix: use three splits. Train on one, fit temperature or the remap on a second split the model never trained on, and report ECE on a third. Use equal count bins with dozens per bin, and freeze the binning before you compare two models.
The Quick Version
- Calibration means stated confidence matches hit rate: answers marked 80% should be right about 80 times in 100.
- One answer proves nothing. Bin by stated confidence, plot hit rate against it, and read distance from the diagonal.
- ECE is the weighted mean of those gaps. In the worked example, 10 at 0.90 with 6 right and 10 at 0.60 with 6 right give 0.5 times 0.30 plus 0.5 times 0.00, which is 0.15.
- Verbalized certainty is a style, not a measurement. Preference tuning rewards decisive tone, so it thins out honest middles and fills high bins with guesses.
- Temperature scaling softens all token probabilities with one fitted knob and keeps ranking unchanged, but it cannot fix topic shift or add missing knowledge.
- Fit any fix on held out prompts and measure on a third split, or the zero you see in the notebook will not survive contact with new questions.