Overfitting vs Underfitting

Every empirical learning problem begins with observed samples generated by an underlying physical law corrupted by measurement noise:

$y = f(x) + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2)$

When you observe ten sample points, your learner faces a fundamental dilemma. Threading every sample point forces the hypothesis to memorize the stochastic noise $\epsilon$, which will never repeat on unseen data. Conversely, ignoring the curve entirely discards the true signal $f(x)$. The goal of machine learning is not zero empirical error on seen points, but minimum expected risk on unseen test points drawn from the same distribution.

Too Stiff Fails

When you fit an overly rigid model class—such as a degree-1 linear function $\hat{y} = w x + b$ through non-linear data—the model suffers from high bias. Because the hypothesis space cannot express curvature, large systematic residuals persist across the entire feature space.

$\text{MSE}_{\text{train}} = 0.45, \quad \text{MSE}_{\text{test}} = 0.48, \quad \text{Gap} = 0.03$

Notice that the generalization gap is tiny ($0.03$), yet both errors are unacceptably high. Collecting more training samples will not resolve this error because model capacity, not sample size, is the structural bottleneck. The model is underfitting.

Sweet Spot Lands

Increasing capacity to match the true data geometry—such as a degree-2 polynomial or a regularized kernel model—allows the hypothesis to bend with the true signal $f(x)$ while resisting individual noise deviations:

$\text{MSE}_{\text{train}} = 0.12, \quad \text{MSE}_{\text{test}} = 0.14, \quad \text{Gap} = 0.02$

Here, the training error and test error converge to a low, stable floor. The minimal gap of $0.02$ confirms that the model has learned the persistent data-generating distribution rather than idiosyncratic sample artifacts. This is the generalization sweet spot.

Too Wiggly Memorizes

When model capacity is unconstrained—such as a degree-9 polynomial with nine free parameters—the optimization algorithm minimizes empirical risk by forcing the curve through every single training point:

$\text{MSE}_{\text{train}} = 0.00, \quad \text{MSE}_{\text{test}} = 0.62, \quad \text{Gap} = 0.62$

Between the training points, the polynomial oscillates wildly with massive Runge-phenomenon excursions. When evaluated on held-out test points, the error explodes to $0.62$. The massive generalization gap of $0.62$ is the unmistakable mathematical signature of overfitting: zero training loss paired with catastrophic test failure.

Curves Tell Truth

Plotting training error and validation error together across model capacity or optimization epochs exposes the classical U-curve of machine learning:

1. Underfitting Regime (Left): Both training and validation error remain high due to severe approximation error (high bias).
2. Optimal Capacity (Center): Validation error reaches its global minimum ($0.14$) at the sweet spot where the model captures signal without tracking noise.
3. Overfitting Regime (Right): Training error monotonically declines toward zero ($0.00$), but validation error climbs steeply ($0.62$) as estimation variance overwhelms the model.

Early stopping halts gradient descent at the validation curve trough, preventing the network from traversing past the sweet spot into the overfitting regime.

Cure Matches Cause

Engineering decisions must be guided strictly by reading the error gap:

• When the gap tops 0.20 (Overfitting): Train error is low but validation error is high. The model has excess variance. Apply $L_2$ weight decay ($\frac{\lambda}{2} \|w\|^2$), inject dropout ($p = 0.2 \dots 0.5$), prune tree depth, or collect more training data to constrain parameter wiggles.
• When both errors stall high (Underfitting): Train and validation errors are both high and close together. The model has excess bias. Increase capacity by adding hidden layers, expanding network width, engineering polynomial feature interactions, or relaxing regularization penalties ($\lambda \to 0$).

Applying the wrong cure—such as regularizing an already underfitting model or deepening an overfitting model—exacerbates the failure mode.

Where It Breaks

A held-out test score is an honest evaluation probe, not an optimization objective. If an engineer repeatedly evaluates against the test set to steer hyperparameter tuning, information leaks from the test set back into the modeling decisions:

$\text{Adaptive Overfitting}: \quad \lim_{k \to \infty} \mathbb{E}[\text{Error}_{\text{test}}] \ll \mathbb{E}[\text{Error}_{\text{production}}]$

This feedback loop contaminates the test vault, producing an artificial test loss of $0.05$ that collapses to $0.58$ in production. To preserve honest generalization estimates, hyperparameters must be tuned exclusively on a validation split, with the sealed test set evaluated strictly once prior to deployment.

The Quick Version

• Ten dots hide one true generating signal corrupted by sample noise.
• Stiff linear models stall near 0.45 on both train and test due to high bias.
• Sweet spot curves balance capacity to land 0.12 train and 0.14 test error.
• Wiggly unconstrained polynomials interpolate train dots at 0.00 but blow up to 0.62 on test.
• The validation error U-curve dips to its minimum before climbing as variance dominates.
• Penalize large gaps with regularization and expand stalled models with capacity.
• Repeated tuning against a test set creates adaptive overfitting and destroys real-world generalization.