Skip to content
AI360Xpert
Core ML
Visual explainer

Model Evaluation

Why you can't trust the training set — the logic of train/val/test splits.

Why training accuracy lies: the test set as a sealed envelope
Why training accuracy lies: the test set as a sealed envelope

A model that just memorises the answers will score 100% on the data it has seen. To measure actual learning, we need a sealed envelope of data it has never seen.

The Three Partitions

Train / validation / test split: what each partition is for
Train / validation / test split: what each partition is for

The dataset is divided into three parts. Train teaches the model. Validation tunes the hyperparameters. Test provides an honest, final evaluation.

The Evaluation Loop

The evaluation loop: fit on train, tune on val, report on test
The evaluation loop: fit on train, tune on val, report on test

The engineer cycles between training and validation, adjusting the model until it performs well. The test set is opened exactly once, at the very end.

Data Leakage

Data leakage: when the future contaminates the past
Data leakage: when the future contaminates the past

Leakage happens when information from the validation or test set accidentally seeps into the training process, leading to artificially high scores.

Where It Breaks

The failure: a great test score that doesn't hold in production
The failure: a great test score that doesn't hold in production

A perfect test score is useless if the test set doesn't reflect the distribution of data the model will encounter in production.

The Quick Version

  • Training accuracy: Lies because models can memorise.
  • Validation set: Used to tune hyperparameters.
  • Test set: Used exactly once for honest evaluation.
  • Leakage: When test info corrupts training.
  • Failure: Test distribution mismatching production.

What to Read Next