Learning Curves

Refit from scratch on growing training sizes and plot both errors. A closed flat gap says buy nothing, while a wide open one says data is the cheapest fix.

Training error rises and validation error falls as rows grow, and with 40 parameters the gap factor falls from 0.10 at 400 rows to 0.013 at 3000 rows

Why Does This Exist?

A vendor offers 30,000 human-judged support replies at 40 seconds of review each. That is 333 hours, most of one person quarter, to score reply helpfulness from 0 to 10. The question is whether the model gets better for it.

The 4,000 replies on hand answer that without spending first. Fit on 200 rows, then 400, 800, up to 3,000, and score each fit on one fixed held-out set. Two curves come out: error where it trained, error where it did not. What matters is not either level but whether both have stopped moving.

One distinction first. A learning curve puts training set size on the horizontal axis and refits from scratch at every point. A training curve puts optimisation steps there from a single fit. Training curves say when to stop training, which belongs to overfitting and underfitting. Learning curves say whether to buy data, which no training curve can, because the data never varied.

This page covers the buying decision. The algebra behind the gap is the bias-variance tradeoff. The sibling plot with data fixed and one setting swept is validation curves. The visual sequence in learning curves visualized draws the same diagnosis frame by frame.

Think of It Like This

Scooping grain before buying the harvest

Forty tonnes of wheat sit in a hopper and you need its average moisture before it is priced. One scoop lands in a wide range and quoting it would be foolish. Ten scoops averaged settle down. One hundred barely budge the number.

The useful part is how you learned that. Ten scoops plotted as they accumulated showed the estimate flattening, so you knew the hundredth scoop was pointless before paying for it. And if the load is genuinely mixed, dry on top and damp at the outlet, no scoop count yields one number for every kernel. That floor is not a sampling problem.

Each scoop is a training size, the settling estimate is validation error, the flat stretch is convergence, and the mixed load is the noise floor. Telling sampling noise from a mixed load is the difference between an afternoon and a quarter.

Where it stops: grain scoops are interchangeable, while training rows carry rare cases, so a curve that looks flat can still hide a subgroup with too few examples to move the average.

How It Actually Works

Training error starts low and rises. With 200 rows and 40 parameters the model bends around those rows, so the error flatters. At 3,000 rows it cannot bend around everything and the error climbs to its honest level. Validation error starts high and falls for the mirror reason: a fit tuned to 200 rows is partly tuned to their noise, and noise does not transfer.

Reading the two endings

Converged, flat, and still bad means both curves have met, neither moves, and the level is unacceptable. Rows 400 to 3,000 changed nothing, so 30,000 more extend a flat line. What remains is a different model, better features, or accepting the number.

A wide gap at the largest size with validation still descending means data is the cheapest fix on the list, costing no expressiveness. Buy the 30,000.

Why the extrapolation is honest

The gap closes on a schedule. For least squares with $d$ parameters, $n$ rows, and noise variance $\sigma^2$ that no feature explains:

\mathbb{E}[\text{training error}] \;\approx\; \sigma^2 \left( 1 - \frac{d}{n} \right)

The movement scales with $d/n$ . With $d = 40$ :

\frac{40}{400} = 0.10 \qquad \frac{40}{3000} \approx 0.013

Nearly all remaining movement happens in the first stretch, which is why the plateau is visible at a tenth of the data under consideration. With $\sigma^2 = 4.0$ the training level climbs from $4.0 \times 0.9 = 3.6$ at 400 rows to about $3.947$ at 3,000 rows, parking against the 4.0 floor the label noise guarantees. The diagram draws both curves with that floor dashed between them and the residual gap bracketed at the right edge.

The protocol that keeps it honest

Spread sizes geometrically, 200, 400, 800, 1600, 3000, because the curvature lives at the small end and even spacing wastes runs. Refit from scratch at every size, since a warm-started fit carries information forward and the curve stops meaning anything. Repeat each size across several subsamples and average, because single runs at small sizes wobble more than the effect being measured. Hold one validation set fixed throughout, falling back to cross-validation schemes when no generous holdout exists, and prefer k-fold cross-validation over a single split when repeats are needed.

Code

The point is the schedule: the gap factor collapsing as rows grow, computed from the formula above.

1params: int = 402noise: float = 4.03for n in (400, 3000):4    gap_factor: float = params / n5    train_level: float = noise * (1 - gap_factor)6    print(n, round(gap_factor, 4), round(train_level, 3))7# -> 400 0.1 3.68# -> 3000 0.0133 3.947

The printout matches the hand computation: 0.10 down to 0.013, training level 3.6 up to 3.947 against the 4.0 floor. Extend the loop to 30,000 and watch the level move by hundredths, which is the whole buying answer in three lines.

Watch Out For

Plotting epochs and pricing data

The plot looks right, two curves converging, someone points at the gap, calls the model data-starved, and 30,000 replies later validation sits where it was. The horizontal axis held training steps with all 4,000 rows used at every point, so the variable in the conclusion never moved.

Symptom: a gap read as hunger for data from a plot that never varied data. Fix: ask whether each point refit from scratch at its size. If not, it is a training curve, right for stopping and for early stopping, wrong for budgeting.

Reading a trend off single runs

One fit per size wobbles down at 200, up at 400, down at 800, and the recommendation depends on which subsample landed where. At small sizes the fold-to-fold spread routinely exceeds the effect being measured.

Symptom: a jagged curve where the last two points decide everything. Fix: ten to twenty repeats per size with mean and spread both plotted, letting the band say whether the curve still descends.

The Quick Version

The horizontal axis is training set size with a fresh refit at every point. Steps there is a different plot answering a different question.
With 40 parameters the gap factor falls from 0.10 at 400 rows to 0.013 at 3000 rows, against a noise floor of 4.0.
Converged, flat, and bad means more data is worthless, so change the model or the features.
A wide open gap with validation still falling means data is the cheapest available fix.
Protocol: geometric sizes, refit from scratch, repeats averaged per size, one fixed validation set.
Single runs at small sizes wobble more than the signal, so bands decide, not points.