Testing ML Systems
You cannot assert exact outputs from a probabilistic model. Test the data going in, the behavior rules that must hold, and the aggregate score on a locked golden set.
Why Does This Exist?
Ordinary software asserts exact answers: add two and two, expect four, ship when it passes. A model never signs that contract. Retrain on the same data with a new seed and a score of 0.62 becomes 0.59, with nothing broken. Worse, the failure that hurts most is silent: the code compiles, the endpoint answers 200, and a shifted input feed turns confident predictions into garbage while every health check stays lit.
So testing splits into three layers. Data checks guard what flows in, behavior rules guard logic that must hold whatever the weights learned, and a locked golden set guards aggregate quality before each release. One term to ground first: precision is the share of flagged items that were right, recall is the share of real items the model found, and their harmonic mean F1 balances the two (see classification metrics).
This page covers pre-release testing. What watches the model after it ships lives in model monitoring, and how runs get recorded for comparison lives in experiment tracking.
Think of It Like This
Examining a student, not a calculator
A calculator either prints 25 for 5 times 5 or it is broken. A student is judged differently. First you check they studied the right book. Then you check they never claim something absurd, like a war fought in the wrong century. Then you give a standardized test and require at least 85 percent.
Data checks are the book. Behavior rules are the never-absurd part. The golden set is the standardized test.
Where the picture stops: a student explains an answer, a model cannot. So the standardized test must be locked away from studying, or the score measures memorization instead of learning.
How It Actually Works
Data tests
Garbage in, garbage out, so the first layer runs before the model ever sees a row. Schema checks confirm the columns exist with the right types. Range checks confirm ages are not negative and incomes sit inside sane bounds. Null checks confirm required fields arrived at all. A violation halts the pipeline immediately, because training or serving on a broken feed wastes everything downstream. These checks are cheap, fast, and catch the largest share of real incidents: feeds change without warning, and the model will happily consume the wreckage.
Behavioral tests
Exact outputs are unknowable, but some rules must hold regardless. Invariance tests change a feature that should not matter and require a steady answer: two identical loan applications differing only in a protected attribute should score within a hair of each other. Directional tests move a feature one way and require the answer to move the right way or stay flat: adding a bathroom should never lower a price estimate. Each rule encodes one piece of domain sense the training loss never stated, and each runs on hand-built pairs rather than sampled data.
Golden sets and worked scores
The last layer is a curated, frozen set of critical and edge cases, run before every promotion. The model must clear aggregate bars and must not regress on named must-pass items. Take a 200-case golden set, 100 positive and 100 negative. The new model returns 88 true positives, 10 false positives, 12 false negatives, 90 true negatives. Precision is , recall is , and F1 is , clearing a 0.85 bar. Ship it? Not yet: the gender invariance pair scores 0.62 and 0.41, a gap of against a tolerance near zero. The aggregate passes and the behavior fails, which is exactly why one layer alone never suffices.
Code
The golden arithmetic plus the invariance gap, recomputed from the fixed counts above.
tp, fp, fn = 88, 10, 12precision = tp / (tp + fp)recall = tp / (tp + fn)f1 = 2 * precision * recall / (precision + recall)print(round(precision, 3)) # -> 0.898print(round(recall, 3)) # -> 0.88print(round(f1, 3)) # -> 0.889
base, mutated = 0.62, 0.41print(round(abs(base - mutated), 2)) # -> 0.21F1 0.889 clears the 0.85 bar while the 0.21 invariance gap fails its own check. Both numbers come from the same release candidate.
Watch Out For
Golden cases leaking into training
Symptom: the golden bar passes with room to spare on every candidate, then the promoted model stumbles on ordinary traffic. The gate kept smiling because the model had studied the test.
Fix: vault the golden set where training code cannot reach it, and audit provenance before each promotion. Any overlap found means rebuilding the set, not explaining it away.
Asserting exact model outputs
Symptom: tests flicker across retrains and seeds, engineers widen tolerances until the checks mean nothing, and real regressions hide inside the noise allowance.
Fix: assert rules and aggregates instead of values. Bound the invariance gap, floor the F1, pin the must-pass items to labels rather than scores, and let A/B measurement settle the close calls on live traffic.
The Quick Version
- Exact output asserts cannot test a probabilistic model, and silent data failures are the costliest kind.
- Data tests check schema, ranges, and missing values before the model runs, and halt the pipeline on violation.
- Behavioral tests pin logic that must hold: invariance under irrelevant changes and sensible direction under relevant ones.
- Golden sets are frozen edge-case collections scored in aggregate, with must-pass items that block promotion.
- One passing layer never clears a release. The worked model scores F1 0.889 yet fails invariance by 0.21.