Early Stopping
Watch validation loss and stop a few epochs after it bottoms out, then roll back to the best checkpoint. Training past that point memorizes noise instead of learning signal.
Why Does This Exist?
You train a fraud classifier for a fixed 30 epochs, then look at the validation curve. Loss falls from 0.92 to 0.31 by epoch 12, then climbs all the way to 0.66 by epoch 30 as the model starts memorizing quirks of the training transactions that never repeat. The weights sitting in your model are from epoch 30, the worst point on the whole curve. Every earlier checkpoint would have scored better on new data.
Guessing a smaller fixed number does not fix it. The bottom of the curve sits at a different epoch for every dataset, architecture, and learning rate, and you only find it by watching the curve while training runs. That is the whole job early stopping does.
Two ideas carry the rest. Validation loss is the error on held out rows the optimizer never sees, so it keeps telling the truth after training loss stops. Overfitting and underfitting is the name for the turn itself: the point where extra training starts hurting. This page covers the stopping rule and the restore step. The theory behind the turn lives in the bias-variance tradeoff, and the curve it watches lives in learning curves.
Think of It Like This
Tasting a sauce as it reduces
You are reducing a sauce. As water cooks off the flavor concentrates and each taste gets better, up to a point, past which it keeps concentrating into something over salted and too thick. You do not set a timer and walk away, because the right moment depends on the pan and the burner today, not on the minutes that worked last time.
So you taste periodically. When two or three tastes in a row stop getting better, you pull the pan off the heat. And the best version was a taste or two ago, not the one in your mouth right now.
The mapping is direct. Each taste is a validation check, the run of flat tastes is the patience counter, pulling the pan is the halt, and the best taste a moment ago is the checkpoint you restore.
Where it stops: you cannot un-reduce a sauce, but you can restore exact old weights. Training gets a rewind button that cooking does not.
How It Actually Works
Training tracks three numbers. The best validation loss seen so far, the epoch that produced it, and a patience counter that counts checks since the last improvement. Each epoch, score validation loss. If it beats the best, save these weights as the new best and reset the counter to zero. If not, add one to the counter. When the counter reaches the patience setting, commonly 5, halt.
Worked example: eighteen epochs, patience five
Watch this validation curve, one number per epoch:
[0.92, 0.78, 0.66, 0.57, 0.50, 0.45, 0.41, 0.38, 0.36, 0.34, 0.33, 0.31, 0.33, 0.38, 0.44, 0.51, 0.58, 0.66]Epochs 1 through 12 each beat everything before, so the best keeps moving and the counter keeps resetting to zero. The minimum is 0.31 at epoch 12. Then epochs 13, 14, 15, 16, and 17 all score worse, so the counter climbs 1, 2, 3, 4, 5. At epoch 17 it hits patience 5 and training halts.
The halt point and the best point differ on purpose. Stopping at the first worse epoch would fire on plain noise, so patience rides out five more epochs to confirm the turn is real. The loop exits holding epoch 17, loss 0.58. The weights you keep come from epoch 12, loss 0.31. The gap you recover by restoring instead of keeping:
That 0.27 is the diagram headline: one curve, a mint marker at epoch 12 where the weights come from, an amber marker at epoch 17 where the loop exits.
Why the restore step is the mechanism, not a nicety
Stopping without restoring throws the benefit away. The model in memory at halt time holds the confirmed-worse weights. Frameworks that implement this seriously save the best checkpoint during the run and reload it at the end, because "whatever the loop held when it exited" is by construction past the optimum. If your best epoch number and your saved checkpoint ever disagree, the restore did not happen.
What it costs
The stopping epoch is a decision made with validation data, exactly like picking a learning rate. Spend that same validation set on more decisions afterward, like choosing between architectures, and its score drifts optimistic from reuse. That is the standard held out reuse problem from hyperparameter tuning, not a special flaw of this method. One clean split per decision keeps the numbers honest.
Code
The point is the trace: rebuild the worked example in plain Python and confirm the loop exits at 17 holding the answer from 12.
def early_stop(val_losses: list[float], patience: int) -> tuple[int, float, int]: best_loss: float = float("inf") best_epoch: int = -1 wait: int = 0 for epoch, loss in enumerate(val_losses, start=1): if loss < best_loss: best_loss = loss best_epoch = epoch wait = 0 else: wait += 1 if wait >= patience: return best_epoch, best_loss, epoch return best_epoch, best_loss, len(val_losses)
val: list[float] = [0.92, 0.78, 0.66, 0.57, 0.50, 0.45, 0.41, 0.38, 0.36, 0.34, 0.33, 0.31, 0.33, 0.38, 0.44, 0.51, 0.58, 0.66]print(early_stop(val, 5)) # -> (12, 0.31, 17)Epochs 1 through 12 strictly decrease, so each resets the wait. Epochs 13 through 17 push it to 5, and the function returns best epoch 12 with loss 0.31 against halt epoch 17. Change patience to 2 and watch it halt at epoch 14 instead, still restoring 12. That gap between halt epoch and best epoch is the patience doing its job.
Watch Out For
Keeping the final weights instead of the best checkpoint
The loop detects the turn correctly, then the code saves whatever is loaded at halt time: epoch 17 in the example, loss 0.58, instead of epoch 12 at 0.31. Nothing errors. The model trains, evaluates, and ships, silently worse by exactly the climb between the two epochs.
Symptom: the logged best epoch and the checkpoint on disk disagree, or reruns with identical settings land on different scores. Fix: save the best checkpoint during the run and reload it explicitly after halting, then assert the restored epoch number equals the logged best before evaluating.
Setting patience so low that noise stops a healthy run
Real validation curves wiggle. Loss ticks up for an epoch or two on batch sampling noise, then resumes falling. Patience 1 or 2 treats that wiggle as the turn and kills a run that had most of its progress ahead of it.
Symptom: training halts within the first few epochs while loss is still trending down overall. Fix: start at patience 5 and scale it to how noisy your metric is, noisier metric means more patience, and plot the curve before trusting any automated halt.
The Quick Version
- Track the best validation loss, the epoch that set it, and a patience counter of checks since the last improvement.
- Halt when the counter reaches patience, which always lands later than the best epoch because patience exists to confirm the turn.
- Restore the best checkpoint explicitly. The weights at halt time are past the optimum by construction.
- In our eighteen epoch run, patience 5 halts at epoch 17 with loss 0.58 and restores epoch 12 at 0.31, recovering 0.27.
- The stopping epoch spends validation information like any tuned setting, so one clean split per decision keeps the score honest.
See the visual explainer on early stopping for the same mechanism step by step.