Model Evaluation

Scoring a model is four decisions and not one number: what it never saw, which mistake you count, where you cut the score, and who it quietly fails for.

Four stacked decisions read bottom to top, and a fraud model that predicts legit always scores 0.99 accuracy with 0.00 recall on 100 frauds in 10000 rows

Why Does This Exist?

A fraud model ships at 99 percent accuracy. Six weeks later losses never move. Someone checks: 100 of 10,000 test rows were fraud and the model flagged nothing at all. Predicting legitimate every time scores 99 percent.

Nobody lied. The number answered a question nobody asked. Evaluation is four decisions and the metric is only the second. Get the split wrong and you measured memory. Get the threshold wrong and you shipped a different product. Skip the slices and you failed the people who matter most while the average smiled.

A held out set is rows the model never fitted, not once and not through a scaler. Any path that lets those rows leak into training makes a too good score exactly that.

Think of It Like This

Two ways to test a driving student

Hand the student the exact route they practiced and they pass. You measured memory of that route, not fitness for the road.

Send them to an unseen road and the score starts to mean something, but you still must choose what counts. Mark only crashes and a student stuck in second gear passes. Mark only speed and a careful driver through a school zone fails.

Even a fair mark hides the turn that matters. A student fine everywhere except unprotected right turns is as risky as one mildly bad everywhere, and the average cannot tell them apart.

New road, right mark, then each turn scored alone. That is split, metric, threshold, slices.

Where it stops: roads do not change because you drove them and examiners do not tune the test until you pass. Data shifts under the model and teams peek at the test until it becomes training, so the split must be locked in a way a road never needs.

How It Actually Works

Read the diagram bottom to top. Each row constrains the one above it.

1 · The split, what has it never seen

Everything above is worthless when this row is wrong. One train test split gives a noisy point. Folds give a mean plus a spread, and the spread says whether a 0.3 point gain is real.

The split must respect structure, not just row count. Rows from one patient or document share one fold or the model memorizes the entity. Time splits forward only. Every fitted step, scaling, encoding, imputation, goes inside the fold. Fitting a scaler on all rows before splitting is the quietest leak in production code.

2 · The metric, which mistake are you counting

A metric prices errors. Missing fraud costs the transaction. Flagging a legit buyer costs an afternoon of trust. Those prices differ, so this is a product choice wearing statistics. Classification metrics all descend from the confusion matrix, the four cell table behind every classification number.

Report a dumb baseline beside the model: majority class, training mean, yesterday value. When the model cannot beat that by more than the fold spread, there is no result yet.

3 · The threshold, where do you cut the score

Most classifiers emit a number between 0 and 1, and 0.5 is a default rather than a decision. Cut at 0.2 and you catch more fraud and annoy more buyers. Cut at 0.8 and the reverse. Same weights, different product. Threshold selection is where the cut is committed from costs.

Worked example: the 99 percent model that catches nothing

Take 10,000 transactions with 100 frauds. The lazy model predicts legitimate always. Right calls are 9,900 of 10,000:

9{,}900 / 10{,}000 = 0.99

True positives are 0 of 100 frauds, so recall is:

0 / 100 = 0.00

Positive predictions are 0, so precision divides by zero and is undefined. Three numbers from one prediction list, one flattering. Change the cut so the model flags 80 frauds plus 40 false alarms and recall jumps to 0.80 while precision lands at 0.67, which shows how one threshold moves both.

4 · The slices, who does it fail for

Break the metric by what the data holds: region, device, language, ticket size. A model at 0.91 overall is often 0.94 on bulk traffic and 0.58 on a slice that is 4 percent of rows and 30 percent of revenue. Averages hide minorities by construction. When a slice matters, it gets its own number and its own alert.

Code

The opening story in arithmetic. No model, no library, just the counting.

1import numpy as np2
3y = np.zeros(10_000, dtype=int)4y[:100] = 1  # 100 frauds hiding in 10,000 transactions5never_flags = np.zeros_like(y)  # answers legitimate always6
7
8def report(y: np.ndarray, pred: np.ndarray) -> str:9    tp = int((pred & y).sum())10    fp = int((pred & (1 - y)).sum())11    fn = int(((1 - pred) & y).sum())12    right = tp + int(((1 - pred) & (1 - y)).sum())13    recall = tp / (tp + fn) if tp + fn else float("nan")14    precision = tp / (tp + fp) if tp + fp else float("nan")15    return f"accuracy {right / y.size:.4f}   recall {recall:.2f}   precision {precision:.2f}"16
17
18print(report(y, never_flags))  # -> accuracy 0.9900   recall 0.00   precision nan

Accuracy 0.99, recall 0.00, precision undefined because no positive prediction exists to judge. The same function on a model that flags 80 frauds with 40 false alarms prints accuracy 0.9880 with recall 0.80, which is why one headline number never settles the choice.

Watch Out For

Tuning on the test set one peek at a time

Nobody trains on test on purpose. It happens slowly: check test, tweak a setting, check again, add a feature, check again. Forty peeks later the test is training and the number is optimistic by an amount nobody can estimate.

Symptom: test wins that vanish on the next month of traffic while validation looked flat. Fix: tune on validation folds, touch the locked test once at the end when nothing will change, and nest the loop when selection and estimation share data.

Optimizing a metric nobody downstream uses

A team lifts AUC from 0.88 to 0.91 while the product shows ten results and reviewers open three. AUC ranks the whole range while the product lives at the top, so the two can move opposite ways for a quarter.

Symptom: offline wins with no online move and debates about why users disagree with math. Fix: write down the action, the cut and the count, then pick the metric that measures that, such as precision at 10 or recall at a fixed alert budget, before the next tuning cycle.

The Quick Version

Evaluation is four decisions: split, metric, threshold, slices. One number answers only the second.
Accuracy on skewed data misleads by design. Always legit on 100 frauds in 10,000 scores 0.99 with recall 0.00.
The split respects entities and time, and every fitted transform lives inside the fold or the score measures leakage.
A dumb baseline plus a fold spread sits beside every score. A win smaller than the spread is not a win.
The 0.5 cut is a default, not a choice. Costs set the threshold and the product changes with it.
Slice the headline metric. The bulk can read 0.94 while a 4 percent slice reads 0.58 and carries the revenue.