Core ML

Early Stopping

Watch validation loss every epoch and stop once it hasn't improved for a set number of checks, then restore the weights from whichever epoch actually scored best — not the one training happened to end on.

Validation loss bottoms out at epoch 12 and climbs afterward; training keeps running for five more epochs to confirm the turn, then restores the weights from epoch 12 rather than epoch 17

Why Does This Exist?

Train a network for a fixed 30 epochs on a fraud-detection dataset and check the validation loss curve afterward: it falls steadily through epoch 12, bottoms out there, then climbs for the remaining 18 epochs as the model starts fitting patterns specific to the training set that don't hold up on new transactions. The weights you actually have, saved at epoch 30 because that's when the loop ended, are the worst checkpoint on the entire curve for anything the model will see in production — worse than epoch 1's, worse than epoch 25's, worse than every single epoch between 1 and 12.

Running fewer epochs doesn't fix this either, unless you know in advance exactly which epoch is the turning point — and that's a number that depends on the data, the architecture, and the learning rate, discoverable only by watching the curve as it happens. Early stopping is the mechanical procedure for finding that turning point during training itself, rather than guessing a fixed epoch count ahead of time and hoping it lands near the right one.

Think of It Like This

Tasting a sauce as it reduces on the stove

A sauce reducing on the stove gets better as water cooks off and the flavor concentrates — up to a point, past which it keeps concentrating into something over-salted and too thick to use. You don't set a timer in advance and walk away, because the exact right moment depends on how much liquid was in the pan and how hot the burner is running today, not on a fixed number of minutes that worked last time.

Instead you taste it periodically. The moment it stops improving — the next taste is no better than the last, or worse — that's roughly your signal to pull it off the heat. Waiting for two or three more tastes to confirm it wasn't a fluke, then pulling it, and remembering that the best taste was actually a moment or two ago, not right now — that's early stopping, applied to dinner instead of a loss curve.

How It Actually Works

The three-part mechanism

Early stopping tracks three things during training: the best validation score seen so far, which epoch produced it, and how many checks have passed since that best score last improved — the patience counter. Every epoch, compute validation loss; if it beats the best-so-far, save this epoch's weights as the new best and reset the patience counter to zero; if it doesn't, increment the counter. Once the counter reaches a chosen threshold — the patience setting, commonly 5 or 10 — stop training.

Why the stop point and the best point aren't the same epoch

Training doesn't halt the instant validation loss ticks upward once; a single worse epoch is often just noise, and stopping on the first uptick would trigger constantly on a genuinely improving run. Patience exists to ride out that noise: training keeps going for several more epochs after the apparent best, confirming the trend really has turned before committing to stop. That means the epoch where training actually halts is always some number of epochs later than the epoch that actually had the best score — the halt point confirms the turn, the saved weights come from before it.

The restore step is not optional

Stopping without restoring the best checkpoint throws away the entire benefit: if training halts at epoch 17 because patience ran out, but epoch 12 had the lowest validation loss, the weights sitting in the model right now are epoch 17's — worse than epoch 12's, by exactly the amount validation loss climbed in those five epochs. The mechanism only delivers what it promises if the actual output is "whichever checkpoint scored best," pulled back explicitly, rather than "whatever the loop was holding when it exited."

What it costs

Early stopping spends information from the validation set to make a decision — which epoch to stop at — the same way choosing a hyperparameter does. Use that same validation set afterward to also pick a learning rate, or a model architecture, or anything else, and the reported validation score starts drifting upward from optimistic reuse, the same overfitting risk that applies to any repeated use of one held-out set for multiple decisions.

Show Me the Code

A synthetic validation curve that bottoms out at epoch 12, with patience 5 confirming the turn before stopping.

1import numpy as np2
3
4def early_stop(val_losses: list, patience: int) -> tuple:5    best_loss, best_epoch, wait = float("inf"), -1, 06    for epoch, loss in enumerate(val_losses):7        if loss < best_loss:8            best_loss, best_epoch, wait = loss, epoch, 09        else:10            wait += 111        if wait >= patience:12            return best_epoch, round(best_loss, 4), epoch13    return best_epoch, round(best_loss, 4), len(val_losses) - 114
15
16rng = np.random.default_rng(0)17epochs = np.arange(30)18val_loss = 1.0 * np.exp(-epochs / 8) + 0.02 * np.maximum(epochs - 12, 0) + 0.3 + rng.normal(0, 0.01, 30)19
20best_epoch, best_loss, stop_epoch = early_stop(list(val_loss), patience=5)21print(f"best epoch: {best_epoch}, best loss: {best_loss}, training halted at epoch: {stop_epoch}")22# -> best epoch: 12, best loss: 0.4999, training halted at epoch: 17

Training runs five epochs past the actual best before patience confirms the turn and halts — the weights that get restored are epoch 12's, not epoch 17's, even though epoch 17 is when the loop exits.

Watch Out For

Saving the final weights instead of restoring the best checkpoint

The single most common way to lose the entire benefit of early stopping: the training loop correctly detects when to stop, but the code saves whatever weights happen to be loaded at that moment — epoch 17 in the example above — rather than reloading the checkpoint from epoch 12. Nothing errors, the model trains and evaluates seemingly fine, and the silent cost is exactly the gap between the two epochs' validation losses, easy to miss unless the best-epoch number and the final saved checkpoint are checked against each other explicitly.

Setting patience so low that noise triggers a stop before real improvement resumes

A validation curve is rarely perfectly smooth; it's common for loss to tick upward for one or two epochs and then resume improving, purely from batch-to-batch variance in what got sampled. Patience set to 1 or 2 stops training on that noise, well before the genuine minimum. There's no universal right value — it depends on how noisy the validation metric is — but a patience that triggers a stop within the first several epochs of an otherwise still-improving run is a signal to check the curve directly rather than trust the automated stop.

The Quick Version

Track the best validation score, which epoch produced it, and a patience counter that increments every epoch without improvement.
Stop once the patience counter reaches its threshold — that halt point is always later than the epoch with the actual best score.
Restoring the best checkpoint, not keeping the final one, is what makes the mechanism deliver its benefit at all.
It's regularization by training duration: cheap, reliable, and it costs no changes to the model or the loss function.
Using the same validation set for early stopping and other tuning decisions accumulates the same optimistic bias repeated reuse always does.