ROC and AUC

AUC measures one thing: the chance a random positive scores above a random negative. That's ranking quality, and it can't see calibration or class imbalance.

Sweeping the cutoff traces true positive rate against false positive rate through 0, 0.75 and 0.5, 0.75, and the area 0.875 is the chance a random positive outranks a random negative

Why Does This Exist?

Two teams hand you a loan risk model each. You cut both scores at $0.5$ , count the outcomes, and model A catches more defaults. Ship A.

Except A may only be more pessimistic. Its scores run higher everywhere, so the same cut flags more of everything. Move B cut down until it catches the same share of defaults, and B flags fewer good customers doing it. Two rulers read at the same tick mark are not the same ruler.

ROC answers by sweeping every cut instead of one. The true positive rate is the share of real defaults flagged, which is recall under another name. The false positive rate is the share of good loans flagged anyway. Plot one against the other and every point is one confusion-matrix. Choosing where to operate on that curve is then a threshold-selection decision, not a property of the curve itself.

Think of It Like This

A blind tasting scored only on the order

Hand a taster twelve unlabelled bottles and ask for a ranking by price. Score every cheap versus expensive pair: did the expensive bottle land ahead. Count that across all pairs.

Every pair right scores $1.0$ . Guessing lands near $0.5$ . Backwards scores near $0$ .

Now what the score cannot see. A taster who prices a 15 coin bottle at about 4 and an 80 coin bottle at about 6 still orders every pair correctly and scores a flawless $1.0$ . Perfect ordering, nonsense numbers. Put that taster in charge of arranging a cellar, never in charge of the price list.

Where it stops: bottles have a true price order while model scores need not even aim at probabilities. A model can rank perfectly yet output numbers no money decision should multiply, which is exactly the calibration blindness below.

How It Actually Works

Start from counts. With $TP$ true positives, $FN$ false negatives, $FP$ false positives and $TN$ true negatives, the rates are $TPR = TP / (TP + FN)$ and $FPR = FP / (FP + TN)$ . Set the cutoff above every score and nothing is flagged, so both rates are $0$ , the bottom left corner. Drop it below every score and everything is flagged, so both are $1$ , the top right. In between sits the curve.

Why the curve only climbs

Lowering a cutoff can only add cases to the flagged set, never remove them. Both the caught positives and the wrongly flagged negatives move up or stay flat, so both rates never decrease. The curve moves only up and right. A model scoring at random pulls the same fraction of positives and negatives above any cut, traces the diagonal, and encloses an area of $0.5$ .

The area is a statement about pairs

The area under the curve equals the probability that a randomly chosen positive scores higher than a randomly chosen negative. No approximation is involved. Thresholds never enter the comparison, so AUC is threshold free, and any increasing rescale of the scores leaves every pairwise order unchanged, so AUC cannot see calibration. Below $0.5$ means the ranking runs inverted, and flipping the sign gives $1 - AUC$ .

Worked example

Take four positives scoring $[0.9, 0.8, 0.7, 0.4]$ and four negatives scoring $[0.6, 0.5, 0.3, 0.2]$ . Count pairs where the positive wins. Scores $0.9$ , $0.8$ and $0.7$ each beat all four negatives, which is $12$ wins. Score $0.4$ beats $0.3$ and $0.2$ only, adding $2$ . Total wins are $14$ out of $4 \times 4 = 16$ pairs, so $AUC = 14 / 16 = 0.875$ .

The sweep tells the same story. Cut at $0.65$ : positives at or above are $3$ of $4$ , negatives $0$ of $4$ , giving $(0, 0.75)$ . Cut at $0.45$ : positives $3$ of $4$ , negatives $2$ of $4$ , giving $(0.5, 0.75)$ . Cut at $0.25$ : positives $4$ of $4$ , negatives $3$ of $4$ , giving $(0.75, 1.0)$ . Those are the dots the diagram joins.

The denominator that hides false alarms

AUC barely notices class imbalance, and on rare event problems that misleads. The false positive rate divides by every negative, a huge count when positives are few. Flag $300$ good customers out of $3916$ and the rate is $300 / 3916 = 0.0766$ , still parked in the good corner, while the review queue drowns in absolute numbers. Nobody staffs a team in units of false positive rate. Under a few percent positives, read the precision-recall-curve alongside.

Code

Counting winning pairs directly, then cubing every score to show what the number ignores. Only the standard library is used.

1pos = [0.9, 0.8, 0.7, 0.4]2neg = [0.6, 0.5, 0.3, 0.2]3
4wins = sum(1 for p in pos for n in neg if p > n)5auc = wins / (len(pos) * len(neg))6print(wins, len(pos) * len(neg))  # -> 14 167print(round(auc, 3))  # -> 0.8758
9pos3 = [s ** 3 for s in pos]10neg3 = [s ** 3 for s in neg]11wins3 = sum(1 for p in pos3 for n in neg3 if p > n)12print(round(wins3 / (len(pos3) * len(neg3)), 3))  # -> 0.875

Cubing changes every score yet swaps no order, so AUC sits still at $0.875$ . That invariance is the strength for comparing rankers and the reason the number cannot bless a shipment or a price.

Watch Out For

Reading a high AUC as permission to ship on rare events

The symptom arrives after launch. AUC near $0.94$ in review, then the queue takes hundreds of cases a week when it was staffed for dozens. Nobody miscomputed AUC.

The false positive rate divided the false alarms by an enormous negative class, so the absolute count never appeared in the headline number. Translate the operating point into counts before committing: alerts per week at this cutoff, and what share are real. That share is precision, and unlike the false positive rate it moves with class balance.

Treating the scores as probabilities because AUC is high

A pricing team sets rates from a $0.93$ AUC model, reasoning that good ranking implies roughly right levels. Months later the book is priced for far more risk than ever materializes, and the loss sits in the margin rather than the ranking.

AUC is blind to every increasing rescale, so a perfectly ordered model can be uniformly off by a large factor. When money multiplies the score, check calibration separately by bucketing predictions and comparing mean predicted risk against observed rates in each bucket.

The Quick Version

Sweep the cutoff from top to bottom and plot true positive rate against false positive rate. Every dot is one confusion matrix.
The curve only climbs up and right, because lowering a cut only adds to the flagged set.
AUC is the chance a random positive outranks a random negative. Four positives against four negatives with $14$ wins give $0.875$ .
AUC is threshold free and blind to increasing rescales, so a badly calibrated model can post an excellent number. Chance is $0.5$ .
Good for asking which model ranks better. Bad for asking whether the queue, the price, or the shipment is safe.