Validation Curves
Learning curves sweep data volume. Validation curves sweep one hyperparameter instead, and the U shape it traces diagnoses over and underfitting directly.
Why Does This Exist?
Learning curves answer would more data help, by sweeping training size with everything else fixed. They never touch the knob itself. This page answers which setting of one knob is right, with data volume fixed instead.
Take ridge regression on four training rows and two features, where only the first feature carries real signal. The strength is the knob. Here is the penalty on large weights, larger means stronger pull toward zero. Too small and the model fits noise in the second feature. Too large and it flattens the real signal too. A learning curve cannot tell you where to set . A validation curve is built for it.
Think of It Like This
Tightening a violin string past the note
Tuning a violin string, you turn the peg and listen. Too loose and the note is flat and dull. The string cannot hold enough tension to sound the pitch cleanly. Turn further and the pitch rises and sharpens. Keep turning past the right point and the string goes shrill, then snaps.
There is one region of the peg travel where the note is true, and it is found by sweeping the whole range and listening for where the sound stops improving and starts getting worse. Regularization behaves the same way against validation error. Too little lets the model chase noise it should ignore, too much discards signal it should keep, and the place worth staying sits between, found by sweeping rather than assuming more or less is always better.
Where it stops. A string snaps past its limit. A model does not snap. It just gets worse gradually on both sides, which is why the picture is a U rather than a breaking point.
How It Actually Works
Two lines on one axis
A validation curve plots two lines against one swept hyperparameter, holding data and all other settings fixed. Training error falls or stays flat as the model gains freedom, since a weaker penalty always lets it fit training rows at least as well. Validation error traces a U. High on the left where the model is too constrained for real signal, high again on the right where it has freedom to fit training noise, lowest in the middle where freedom matches what the data supports.
The gap between the lines is the same overfitting signal overfitting and underfitting reads off a training step axis. Training far below validation means memorizing rather than generalizing. Here the axis is a hyperparameter, so the diagnosis carries a fix. Move the knob back toward the U minimum.
Worked example
Ridge on four rows with two features, in . Training errors are , , , and , climbing steadily as grows. Validation errors are , , , and , falling to a minimum at then rising again. That minimum, not either end, is the setting to keep. The diagram above draws this U with its low point at , and the code below computes the same four pairs.
How to sweep and what to keep
Sweep on a log scale when the natural scale is multiplicative, such as regularization strength or learning rate, because a linear sweep bunches resolution at one end. Read the minimum of the validation line, never the training line, which has no useful minimum past memorization. If the minimum sits at the edge of the range, the range was too narrow, not the answer. A one knob validation curve explains why a setting works. A full search over many knobs in hyperparameter tuning finds the best combination. Run the curve first for the knob that matters most.
Code
Ridge strength swept on tiny fixed data, with a closed form for two weights so no solver is needed.
Xtr = [[1, 1], [2, 0], [3, 1], [4, 0]]ytr = [1.0, 2.5, 2.5, 4.5]Xva = [[5, 1], [6, 0]]yva = [5.0, 6.0]
def ridge_mse(alpha: float) -> tuple[float, float]: a11 = sum(r[0] * r[0] for r in Xtr) + alpha; a22 = sum(r[1] * r[1] for r in Xtr) + alpha a12 = sum(r[0] * r[1] for r in Xtr); b1 = sum(r[0] * y for r, y in zip(Xtr, ytr)); b2 = sum(r[1] * y for r, y in zip(Xtr, ytr)) det = a11 * a22 - a12 * a12; w0 = (b1 * a22 - b2 * a12) / det; w1 = (a11 * b2 - a12 * b1) / det tr = sum((r[0] * w0 + r[1] * w1 - y) ** 2 for r, y in zip(Xtr, ytr)) / len(ytr) va = sum((r[0] * w0 + r[1] * w1 - y) ** 2 for r, y in zip(Xva, yva)) / len(yva) return tr, va # train cost of freedom, validation price of it
for a in [0.01, 1.0, 10.0, 100.0]: print(round(a, 2), [round(v, 2) for v in ridge_mse(a)])# -> 0.01 [0.09, 0.23]# -> 1.0 [0.12, 0.04]# -> 10.0 [0.69, 1.38]# -> 100.0 [5.04, 17.45]Training climbs to while validation dips to at then climbs to . The U minimum is the keeper.
Watch Out For
Reading the training line minimum instead of the validation one
Symptom: the plot keeps falling on the training line toward the most permissive setting, and the team ships that end point as still improving. Validation already turned upward two grid points earlier.
Fix: decide from the validation line only. The training line exists as the comparison that reveals overfitting, never as the selector. Mark the validation minimum explicitly before discussing the plot.
Sweeping too narrow and mistaking an edge for a minimum
Symptom: a sweep from to bottoms at , the largest value tried, and the report calls it the answer. Everything past stays untested.
Fix: whenever the keeper sits at either edge, widen the range and rerun before trusting the curve. A minimum at an edge proves the search was too narrow, not that the search is done.
The Quick Version
- A validation curve sweeps one hyperparameter with training data fixed, while a learning curve sweeps data volume with the model fixed.
- Training error falls or flattens as freedom grows. Validation error traces a U.
- The worked ridge training errors are , , , and , while validation is , , , and .
- The validation minimum at is the setting to keep, never the training line.
- Sweep multiplicative knobs on a log scale, and widen the range when the keeper lands at an edge.