Classification Metrics

Precision prices a false alarm, recall prices a miss, and one threshold moves both. Every number you report is a claim about which mistake you'd rather make.

Two views of 210 catches in 1,100 flags and 300 real takeovers give precision 0.19 and recall 0.70.

Why Does This Exist?

A support inbox takes 40,000 tickets a week. About 300 are account takeovers, a stranger inside with the recovery address changed and money moving. Those need a security engineer within the hour.

Your model flags 1,100 tickets and catches 210 of the 300.

Is that good? One number cannot say, because two questions hide inside and they disagree. Of the 1,100 you flagged, 210 were real, so 0.19. Of the 300 real ones, you caught 210, so 0.70.

Those are precision and recall. They stay separate because they price different mistakes. A false alarm costs eleven minutes of engineer time on a password reset dressed as an intrusion, plus the odd frozen account that belonged to someone innocent. A miss gives the attacker more days inside. That is why you will never ship one without the other and its threshold.

Mechanically both are one count over two totals, everything you flagged and everything that really was a takeover. That split is the diagram. The four counts themselves live in the confusion matrix, which assumes someone checked those tickets by hand to make ground truth.

This page turns those counts into precision, recall, F1 and F-beta. The precision-recall curve sweeps every threshold at once. Threshold selection is where you commit to one cut from costs. ROC and AUC give the ranking view that cannot see calibration, and model evaluation puts the metric beside the split, the threshold and the slices. The precision versus recall comparison settles which side to favour when you must pick.

Think of It Like This

The mesh size on a fishing net

You are after mackerel, and your net has one setting that matters, how wide the holes are.

Pull it tight and almost nothing escapes. You land every mackerel in the water, plus seaweed, jellyfish, bottles and undersized fish you throw back. Huge haul, mostly junk.

Open the mesh and the haul comes back clean, nearly all keepers. But most of the mackerel swam through, and you will never know how many.

One hole size does both jobs, so you cannot fix one side without moving the other. And each number alone is trivially faked. Strain the whole sea and you caught 100 percent of the mackerel. Refuse to fish and everything in your empty net is a mackerel.

Where it stops: the net cannot price your mistakes. It does not know a missed mackerel costs dinner while a false haul costs an afternoon sorting junk. Fishermen set mesh from prices and quotas, and you set thresholds the same way, from costs.

How It Actually Works

Start from the four counts in the confusion matrix. Write $TP$ for catches that were real, meaning flagged and truly a takeover. Write $FP$ for false alarms, meaning flagged but clean. Write $FN$ for misses, meaning real but passed. Write $TN$ for clean tickets correctly passed. Our week gives $TP = 210$ , $FP = 890$ , $FN = 90$ and $TN = 38,810$ . You can check the totals yourself, flagged is $210 + 890 = 1,100$ and really positive is $210 + 90 = 300$ .

Precision asks, of what you flagged, how much was real. Recall asks, of what was real, how much you flagged. Same numerator, different denominator:

P = \frac{TP}{TP + FP} = \frac{210}{1,100} \approx 0.19

R = \frac{TP}{TP + FN} = \frac{210}{300} = 0.70

$P$ is precision and $R$ is recall. The diagram draws this twice. On the left the brighter column is everything flagged, 1,100 tickets, and 210 over that column is 0.19. On the right the brighter row is everything really positive, 300 tickets, and 210 over that row is 0.70. The accented cell is in both, because it is the same 210 catches.

Two baselines show why neither stands alone. Flag all 40,000 tickets and recall is exactly 1.0 while precision falls to $300 / 40,000 = 0.0075$ . Flag nothing and recall is 0 while precision has no denominator at all. Both are one line of code away, which is why a report without its threshold tells you nothing.

Accuracy hides the same week in plain sight:

\frac{TP + TN}{40,000} = \frac{39,020}{40,000} = 0.9755

That reads 0.98 while 90 takeovers walk out. The F1-score definition names the fix, you must price the two errors separately.

Why F1 leans low and what beta changes

F1 is the harmonic mean of $P$ and $R$ :

F_1 = \frac{2PR}{P + R} = \frac{420}{1,400} = 0.30

The second form uses counts directly, $2TP$ over $2TP + FP + FN$ , so $420$ over $1,400$ . Harmonic sits near the smaller input and the larger cannot rescue it. Take precision 0.99 with recall 0.01, a model that flags three tickets a week and sleeps through nearly every intrusion. Ordinary averaging gives 0.50, a respectable score for something close to worthless. F1 gives $0.0198$ .

But F1 hides a choice, that both errors matter equally. For a takeover queue that is false. F-beta makes the weight a parameter:

F_\beta = \frac{(1 + \beta^2) PR}{\beta^2 P + R}

Here $\beta$ says how many times more recall counts than precision, and the weighting goes as $\beta^2$ . At $\beta = 1$ you are back to F1. At $\beta = 2$ recall carries four times the weight:

F_2 = \frac{5 \times 210}{5 \times 210 + 4 \times 90 + 890} = \frac{1,050}{2,300} \approx 0.46

Choosing $\beta$ is a costing exercise, not a modelling one. Engineer minutes on one side, what your incident write ups say a takeover costs on the other. Report $F_\beta$ beside raw precision and recall, never instead of them.

What averaging hides with more than two classes

Same inbox, bigger job, four queues instead of yes or no. Precision and recall go per class, so a headline needs an average, and three conventions disagree.

Macro gives every class one vote. Three queues scoring F1 0.90, 0.60 and 0.30 average to 0.60, so a queue with 3 percent of traffic counts as much as one with 86 percent.
Micro pools all counts first, then divides once. Since each ticket is one prediction and one truth, pooled precision and recall come out identical, so micro F1 equals accuracy and follows the biggest class.
Weighted averages by frequency. With shares 0.86, 0.11 and 0.03 the same three scores give $0.86 \times 0.90 + 0.11 \times 0.60 + 0.03 \times 0.30 = 0.849$ , about 0.85. Same predictions, macro 0.60 and weighted 0.85.

Use macro when the rare queue is the reason you built the thing. Use micro or weighted when bulk throughput is the product. Either way, publish per class numbers first. An average that moves from 0.60 to 0.85 by choice of weights cannot be the only number on the card.

Code

The point is the check, same 210 catches recomputed from counts, no library needed.

1def precision_recall_f1(tp: int, fp: int, fn: int) -> tuple[float, float, float]:2    precision: float = tp / (tp + fp)3    recall: float = tp / (tp + fn)4    f1: float = 2 * precision * recall / (precision + recall)5    return precision, recall, f16
7
8tp: int = 2109fp: int = 89010fn: int = 9011p: float12r: float13f1: float14p, r, f1 = precision_recall_f1(tp, fp, fn)15print(f"{p:.3f} {r:.3f} {f1:.3f}")  # -> 0.191 0.700 0.30016print(f"{2 * tp}/{2 * tp + fp + fn} = {f1:.2f}")  # -> 420/1400 = 0.30

The printout matches the hand computation, 210 over 1,100 is 0.19, 210 over 300 is 0.70, and 420 over 1,400 is 0.30.

Watch Out For

Shipping F1 as though it were the neutral option

The symptom is a model card with one number on it and no threshold beside it. Someone asks why the takeover queue keeps missing intrusions, and the answer is a default accepted months ago, $\beta = 1$ with precision and recall equally weighted. Nobody chose it. It was the shape of the function call.

F1 is a position, not a baseline. Price both mistakes, set $\beta$ from that ratio, and freeze the threshold on validation data. Then report that threshold beside raw precision and recall. One number cannot tell a reader which way the model fails, and threshold selection is where that choice belongs.

Quoting accuracy as evidence the rare class works

The symptom is a dashboard reading 0.98 accuracy across 40,000 tickets and the review moves on. Then takeovers turn up in the billing queue, because 90 misses hide inside 38,810 correct passes. Nothing was misreported. Accuracy divides by everything, so a class holding under 1 percent of rows contributes under 1 percent of the evidence.

Report per class precision and recall, always, and put macro beside micro so the gap shows. Then check the rare class is measurable at all. With 300 takeovers split over five folds you get 60 a fold, so six tickets move recall ten points. Stratified sampling keeps those folds comparable.

The Quick Version

Precision is 210 catches over 1,100 flags, about 0.19. Recall is the same 210 over 300 real takeovers, exactly 0.70.
One threshold moves precision and recall in opposite directions, and either alone can be maxed by flagging everything or nothing.
F1 is the harmonic mean, so it sits near the weaker half. Precision 0.99 with recall 0.01 scores 0.02, not 0.50.
F-beta makes the equal weight explicit, with recall weighted by beta squared. Beta 2 lifts this week from F1 0.30 to F2 0.46.
Macro gives each class one vote while micro pools counts and equals accuracy, so the same predictions can read 0.60 or 0.85 by choice of average.
Accuracy 0.98 on 40,000 tickets still misses 90 of 300 takeovers, which is why rare problems report precision, recall and their threshold together.