Cross-Validation Schemes

Why Does This Exist?

You held out 20 percent for validation, scored 0.847, and now you are choosing between models that differ by 0.3 points. How much would 0.847 move on a different 20 percent? You have no idea.

Cross-validation answers that. Rotate the validation block through the data, fit K times, and you get K scores from the same rows: a mean and a spread.

The spread is the half everyone drops from the report, and the more useful half. At a fold standard deviation of 2 points, a 5-fold mean has standard error $2/\sqrt{5} = 0.89$, so a 0.3-point difference is a third of a standard error. It means nothing.

This page is about partitioning the data honestly. Picking a model with those partitions belongs to tuning, and the single split everything here generalizes is in train validation test splits.

The one idea you need first

A validation score from one split is a random number. Change which rows land in the 20 percent and the score moves, because some rows are harder than others. That sampling wobble is what the spread across folds measures. If that idea is new, read the splits page first, then come back. The rest of this page assumes you know what a held-out set is.

Think of It Like This

How It Actually Works

Each rung fixes something the one below it gets wrong. Climb until your data stops breaking the assumption. Here $K$ is the number of folds, $n$ is the number of rows, and each fold validates a different $n/K$ rows exactly once.

The ladder

• K-fold. Split into $K$ parts. Train on $K-1$, validate on the one left out, rotate. Small $K$ is pessimistic because each model sees less data. Large $K$ has low bias but the folds correlate because their training sets overlap heavily. Five or ten is the usual pick.
• Stratified K-fold. Each fold keeps the class proportions of the whole. This is the default for classification. At 2 percent positives a plain fold can arrive with almost no positives, and that score is noise the mean then absorbs. See imbalanced data.
• Grouped K-fold. No entity appears on both sides of a fold boundary. Patient, user, device, session: split one across folds and the model memorizes it in training and recognizes it in validation. You measured recall of memory. A stratified-grouped variant does both jobs approximately.
• Leave-one-out and repeats. $K = n$ gives the lowest bias and a noisy estimate, because each fold scores one row and the $n$ models are nearly identical. Repeated K-fold reruns 5-fold under several shuffles and pools the scores.
• Time-series split. Never shuffled. Training always sits earlier than validation. An expanding window grows from the start of the series. A rolling window keeps training length fixed and slides forward. See time series data.
• Purged and embargoed. For labels that span time. If the label is the price move over the next 5 days, Monday and Wednesday rows share three days of outcome, so rows near a boundary correlate. Purging drops training rows whose label window overlaps validation. An embargo drops a further gap after that.
• Nested. The outer loop estimates and the inner loop tunes, on the outer loop training rows only, because one set of folds cannot do both honestly. Five outer folds, five inner folds, a 20-point grid: 505 fits.

Every fitted step refits inside the fold

This silently invalidates more results than any wrong choice of $K$.

A scaler learns a mean. An imputer learns a median. An encoder learns a category list. Feature selection learns which columns survive, and target encoding learns an average of the label. All of that is fitted state, so all of it refits inside every fold on that fold training rows only. A Pipeline passed to the cross-validator does that for you. A hand-rolled loop usually does not, because the tempting place for preprocessing is above the loop.

Worked example

Twenty rows, indices 0 to 19, five folds. Unshuffled, the validation blocks are [0, 1, 2, 3], [4, 5, 6, 7], [8, 9, 10, 11], [12, 13, 14, 15], [16, 17, 18, 19]. Now make those rows five users with four consecutive rows each: user 0 owns 0 to 3, user 1 owns 4 to 7, and so on. Each fold holds exactly one user, because the table was sorted that way. That is luck, not design.

Shuffle with a fixed order and it falls apart. The first block becomes [2, 4, 6, 19], with users 0, 1 and 4. The user counts across the five folds run [3, 4, 3, 2, 3]. User 1 rows 4 and 6 validate while rows 5 and 7 train, so the model recognizes its own training users at scoring time.

Code

1import numpy as np2
3n, k = 20, 54groups: np.ndarray = np.repeat(np.arange(5), 4)  # 5 users, 4 rows each5
6def folds(order: np.ndarray) -> list[np.ndarray]:7    return list(np.array_split(order, k))  # 20 rows in 5 blocks of 48
9def users_per_fold(parts: list[np.ndarray]) -> list[int]:10    return [int(np.unique(groups[p]).size) for p in parts]11
12ordered = folds(np.arange(n))13print([p.tolist() for p in ordered[:2]])  # -> [[0, 1, 2, 3], [4, 5, 6, 7]]14print(users_per_fold(ordered))  # -> [1, 1, 1, 1, 1]15mixed = np.array([2, 4, 6, 19, 0, 5, 10, 17, 1, 3, 8, 12, 11, 13, 14, 15, 7, 9, 16, 18])16shuffled = folds(mixed)17print(sorted(shuffled[0].tolist()))  # -> [2, 4, 6, 19]18print(users_per_fold(shuffled))  # -> [3, 4, 3, 2, 3]

One user per fold before the shuffle, three or four after. The default shuffle is exactly what breaks grouped data, which is why groups get their own splitter.

Watch Out For

The Quick Version

• One validation split gives one noisy number. Cross-validation gives a mean and a spread, and the spread is the half that matters: at a fold standard deviation of 2 points, a 5-fold mean has standard error 0.89.
• K-fold validates every row once. $K$ of 5 or 10 is usual. Small $K$ is pessimistic and large $K$ correlates the folds. Leave-one-out is the $K=n$ limit: low bias, noisy estimate.
• Stratified is the default for classification. Grouped is mandatory when rows share an entity.
• Time-series splits never shuffle: training sits earlier than validation, expanding or rolling.
• Purged and embargoed splits handle labels that span time, where adjacent rows share outcome.
• Nested cross-validation tunes inside and estimates outside. Five by five on a 20-point grid is 505 fits.
• Every fitted step refits inside the fold: scaler, imputer, encoder, selector, target encoder. That is what a Pipeline is for.
• Twenty rows from five users hold one user per fold unshuffled, then three or four per fold after shuffling.

What to Read Next

• Train, Validation and Test Splits
• Data Leakage
• Class Imbalance