Skip to content
AI360XpertBeta

Model Selection and Validation

Pick the winner on one slice of data, then report its score on a slice nobody touched during development. One slice for choosing, one slice for telling the truth.

Validation scores pick the forest at 0.86 over 0.787 and 0.827, and the test set is touched once to report 0.833, so no choice leaks into the final number
Validation scores pick the forest at 0.86 over 0.787 and 0.827, and the test set is touched once to report 0.833, so no choice leaks into the final number

Why Does This Exist?

A team trains three classifiers for fraud review, checks all three on the test set, ships the forest because it scored highest, and watches live accuracy land three points below the reported number. Nobody mislabeled anything. The test set helped make a decision, so its number flattered the winner. Whatever luck the forest had on those particular rows got counted as skill.

That is the whole reason for the split. Model selection asks which candidate to ship. Validation asks what score to believe about future data. One question needs data you are allowed to consult freely; the other needs data you consult exactly once. Answer both from the same rows and the second answer is optimistic by construction.

This page covers the three-way split and the selection rule. The full procedure for selecting and tuning together lives on the nested cross-validation page, and splits for time-ordered data live on the time-series cross-validation page.

The one idea you need first. A model can memorize its training rows and still fail on new ones, which is overfitting and underfitting. That gap is why a score computed on training data can never be the basis for choosing between candidates.

Think of It Like This

A baking contest with a semifinal and a final

A baking contest runs in two tastings. For weeks the bakers practice in their own kitchens, trying recipe after recipe. Then a semifinal tasting picks which cake goes to the final. Then fresh judges, who tasted nothing before, score that one cake exactly once, and the score they give is the official result.

The semifinal is allowed to be opinionated: it exists to choose. The final exists to measure, so its judges must be new and get one tasting. If a baker could watch the final judges react and swap cakes mid-contest, the winning score would stop meaning anything. It would measure the swapping, not the baking.

Where the analogy stops: contest judges have personal tastes, while data splits are random draws from the same kitchen. And real leakage is sneakier than swapping a cake. It looks like checking one more tweak against the final score, which feels harmless and does the same damage.

How It Actually Works

The three-way split

Take a labeled dataset of nn rows and cut it into three disjoint parts before fitting anything:

  • Training set, about 60 to 70 percent: every candidate model fits here, as many times as you like.
  • Validation set, about 10 to 20 percent: candidates are compared here. Accuracy on these rows is the selection score, and the winner is the argmax, the candidate with the highest validation score.
  • Test set, about 10 to 20 percent: the winner is evaluated here exactly once, and that number is the reported performance.

The validation set absorbs all the peeking. Every comparison, every tie-break, every "let me try one more setting" consults validation, never test. The test set stays sealed until the choice is final, which is what keeps the final number honest.

Worked example

Say n=1000n = 1000 rows, split into 700 for training, 150 for validation, and 150 for test. Three candidates train on the 700, then each predicts the 150 validation rows:

  • Logreg gets 118 right: 118/150=0.787118 / 150 = 0.787.
  • Forest gets 129 right: 129/150=0.860129 / 150 = 0.860.
  • Boosted trees get 124 right: 124/150=0.827124 / 150 = 0.827.

The forest wins on validation and nothing else is consulted. Then, once, the forest predicts the 150 test rows and gets 125 right: 125/150=0.8333125 / 150 = 0.8333, reported as 0.833. That 0.833 is believable precisely because the test rows never influenced the pick. Notice the winner's validation score (0.860) sits above its test score (0.833). Winners usually look slightly worse on fresh data, and that gap is normal, not a bug.

When 150 rows are not enough

A single validation slice of 150 rows is noisy, and with little data it gets worse. The standard replacement is kk-fold cross-validation run on the training data only: split train into kk equal folds (5 is the common default), fit each candidate kk times with a different fold held out, and average the kk held-out scores. Pick the best average, refit that candidate on all the training data, and evaluate once on the test set. When selection and hyperparameter tuning both need doing, one loop conflates them, and that is the nested procedure on the nested cross-validation page.

Code

The point of this snippet is the selection rule itself: the pick is an argmax over validation scores, and the test score is computed once, after.

val_correct: dict[str, int] = {"logreg": 118, "forest": 129, "boosted": 124}n_val: int = 150
for name, correct in val_correct.items():    print(name, round(correct / n_val, 3))# -> logreg 0.787# -> forest 0.86# -> boosted 0.827
print("test", round(125 / n_val, 3))# -> test 0.833

The forest is picked because 0.86 beats 0.827 and 0.787 on validation. The test line runs a single time, after the choice is final.

Watch Out For

Peeking at the test set one tweak at a time

The symptom is a test score that climbs a point every week while live performance sits still. Each tweak was small and each check felt read-only, but every decision the test score influenced leaked test information into the model. After a month of tweaks, the test set is a second validation set wearing a disguise.

The fix is procedural, not mathematical. Write down the date you first evaluate on test, evaluate exactly once, and freeze the model when you do. If the number disappoints, the remedy is more training data or a better candidate, evaluated on a fresh split, never another tweak against the same test rows.

Shuffling time-ordered rows into random folds

The symptom is cross-validation at 0.92 and live performance at 0.78. Random folds trained the model on future rows and validated it on past ones, so validation rewarded patterns that only exist with hindsight. Fraud, demand, and failures all drift with time, and a shuffled split hides the drift.

Split by time instead: train on earlier rows, validate on later ones, always. The time-series cross-validation page shows the expanding-window version that keeps the future strictly out of training.

The Quick Version

  • Training rows fit candidates, validation rows choose between them, and test rows are evaluated exactly once to produce the reported score.
  • The winner is the argmax of validation accuracy: in the worked example the forest at 0.860 beats 0.827 and 0.787, then reports 0.833 on test.
  • A validation score above the test score is normal. Winners look slightly worse on fresh data.
  • With little data, replace the single validation slice with kk-fold cross-validation on the training data only, then test once.
  • Every extra peek at the test set turns it into a second validation set, and the reported number becomes optimistic.