Skip to content
AI360XpertBeta

Validation Curves

Learning curves sweep data volume. Validation curves sweep one hyperparameter instead, and the U shape it traces diagnoses over and underfitting directly.

Training error falls as regularization weakens, but validation error traces a U with its minimum where model freedom matches what the data supports
Training error falls as regularization weakens, but validation error traces a U with its minimum where model freedom matches what the data supports

Why Does This Exist?

Learning curves answer would more data help, by sweeping training size with everything else fixed. They never touch the knob itself. This page answers which setting of one knob is right, with data volume fixed instead.

Take ridge regression on four training rows and two features, where only the first feature carries real signal. The strength alphaalpha is the knob. Here alphaalpha is the penalty on large weights, larger means stronger pull toward zero. Too small and the model fits noise in the second feature. Too large and it flattens the real signal too. A learning curve cannot tell you where to set alphaalpha. A validation curve is built for it.

Think of It Like This

Tightening a violin string past the note

Tuning a violin string, you turn the peg and listen. Too loose and the note is flat and dull. The string cannot hold enough tension to sound the pitch cleanly. Turn further and the pitch rises and sharpens. Keep turning past the right point and the string goes shrill, then snaps.

There is one region of the peg travel where the note is true, and it is found by sweeping the whole range and listening for where the sound stops improving and starts getting worse. Regularization behaves the same way against validation error. Too little lets the model chase noise it should ignore, too much discards signal it should keep, and the place worth staying sits between, found by sweeping rather than assuming more or less is always better.

Where it stops. A string snaps past its limit. A model does not snap. It just gets worse gradually on both sides, which is why the picture is a U rather than a breaking point.

How It Actually Works

Two lines on one axis

A validation curve plots two lines against one swept hyperparameter, holding data and all other settings fixed. Training error falls or stays flat as the model gains freedom, since a weaker penalty always lets it fit training rows at least as well. Validation error traces a U. High on the left where the model is too constrained for real signal, high again on the right where it has freedom to fit training noise, lowest in the middle where freedom matches what the data supports.

The gap between the lines is the same overfitting signal overfitting and underfitting reads off a training step axis. Training far below validation means memorizing rather than generalizing. Here the axis is a hyperparameter, so the diagnosis carries a fix. Move the knob back toward the U minimum.

Worked example

Ridge on four rows with two features, alphaalpha in [0.01,1.0,10.0,100.0][0.01, 1.0, 10.0, 100.0]. Training errors are 0.090.09, 0.120.12, 0.690.69, and 5.045.04, climbing steadily as alphaalpha grows. Validation errors are 0.230.23, 0.040.04, 1.381.38, and 17.4517.45, falling to a minimum at alpha=1.0alpha = 1.0 then rising again. That minimum, not either end, is the setting to keep. The diagram above draws this U with its low point at alpha=1.0alpha = 1.0, and the code below computes the same four pairs.

How to sweep and what to keep

Sweep on a log scale when the natural scale is multiplicative, such as regularization strength or learning rate, because a linear sweep bunches resolution at one end. Read the minimum of the validation line, never the training line, which has no useful minimum past memorization. If the minimum sits at the edge of the range, the range was too narrow, not the answer. A one knob validation curve explains why a setting works. A full search over many knobs in hyperparameter tuning finds the best combination. Run the curve first for the knob that matters most.

Code

Ridge strength swept on tiny fixed data, with a closed form for two weights so no solver is needed.

Xtr = [[1, 1], [2, 0], [3, 1], [4, 0]]ytr = [1.0, 2.5, 2.5, 4.5]Xva = [[5, 1], [6, 0]]yva = [5.0, 6.0]

def ridge_mse(alpha: float) -> tuple[float, float]:    a11 = sum(r[0] * r[0] for r in Xtr) + alpha; a22 = sum(r[1] * r[1] for r in Xtr) + alpha    a12 = sum(r[0] * r[1] for r in Xtr); b1 = sum(r[0] * y for r, y in zip(Xtr, ytr)); b2 = sum(r[1] * y for r, y in zip(Xtr, ytr))    det = a11 * a22 - a12 * a12; w0 = (b1 * a22 - b2 * a12) / det; w1 = (a11 * b2 - a12 * b1) / det    tr = sum((r[0] * w0 + r[1] * w1 - y) ** 2 for r, y in zip(Xtr, ytr)) / len(ytr)    va = sum((r[0] * w0 + r[1] * w1 - y) ** 2 for r, y in zip(Xva, yva)) / len(yva)    return tr, va  # train cost of freedom, validation price of it

for a in [0.01, 1.0, 10.0, 100.0]:    print(round(a, 2), [round(v, 2) for v in ridge_mse(a)])# -> 0.01 [0.09, 0.23]# -> 1.0 [0.12, 0.04]# -> 10.0 [0.69, 1.38]# -> 100.0 [5.04, 17.45]

Training climbs 0.090.09 to 5.045.04 while validation dips to 0.040.04 at alpha=1.0alpha = 1.0 then climbs to 17.4517.45. The U minimum is the keeper.

Watch Out For

Reading the training line minimum instead of the validation one

Symptom: the plot keeps falling on the training line toward the most permissive setting, and the team ships that end point as still improving. Validation already turned upward two grid points earlier.

Fix: decide from the validation line only. The training line exists as the comparison that reveals overfitting, never as the selector. Mark the validation minimum explicitly before discussing the plot.

Sweeping too narrow and mistaking an edge for a minimum

Symptom: a sweep from alpha=1alpha = 1 to alpha=10alpha = 10 bottoms at alpha=10alpha = 10, the largest value tried, and the report calls it the answer. Everything past 1010 stays untested.

Fix: whenever the keeper sits at either edge, widen the range and rerun before trusting the curve. A minimum at an edge proves the search was too narrow, not that the search is done.

The Quick Version

  • A validation curve sweeps one hyperparameter with training data fixed, while a learning curve sweeps data volume with the model fixed.
  • Training error falls or flattens as freedom grows. Validation error traces a U.
  • The worked ridge training errors are 0.090.09, 0.120.12, 0.690.69, and 5.045.04, while validation is 0.230.23, 0.040.04, 1.381.38, and 17.4517.45.
  • The validation minimum at alpha=1.0alpha = 1.0 is the setting to keep, never the training line.
  • Sweep multiplicative knobs on a log scale, and widen the range when the keeper lands at an edge.