Cross-Validation
K-fold as a mechanism for honest error estimation — why the test set must stay sealed.
A single train/validation split relies on luck. If the validation split accidentally contains all the easiest or hardest examples, the score will wildly misrepresent the model's true capability.
K-Fold Cross-Validation
K-fold fixes this by dividing the data into K equal chunks (folds). The model trains on K-1 folds and validates on the remaining one. It repeats this until every fold has been the validation set exactly once, then averages the scores.
Stratified Folds
With imbalanced data, random folds might contain zero positive examples. Stratified K-fold ensures every fold maintains the exact same class distribution as the overall dataset.
The Sealed Test Set
Cross-validation is a substitute for the validation split, not the test set. Because you use CV scores to tune hyperparameters, the model indirectly learns from those folds. You still need a completely untouched test set for the final report.
Where It Breaks
The most common failure is data leakage. If you scale your data or impute missing values before splitting into folds, information from the validation fold leaks into the training step. Every transformation must happen inside the CV loop.
The Quick Version
- The Problem: Single splits are highly sensitive to randomness.
- K-Fold: Rotate the hold-out set so every row gets evaluated.
- Stratified: Forces class ratios to stay constant across folds.
- Sealed Test Set: CV replaces validation, you still need a final holdout.
- Failure: Pre-processing before CV leaks information.