Foundations
Bias-Variance Tradeoff
How model complexity drives a tradeoff between capturing the underlying signal (bias) and overreacting to noise (variance).
Stage 1 of 5: The Data
Degree 1 fitted to 20 training points. Training error 0.093, validation error 0.098.
- Training sample
- Held-back validation sample
How many training points the model gets to learn from.
Twenty training points and twenty held-back validation points, drawn from one hidden curve plus noise. How many points you have is a real parameter here, not a backdrop — it decides everything that follows.
- sample size
- degree
- fitted curve
- training error
- validation error
Check your understanding
Why doesn't a model that perfectly memorizes the training data generalize well? See exactly how model complexity trades off between capturing the broad signal and obsessing over the noise.
Overfitting and Underfitting
When building a model, the goal is to uncover the underlying pattern holding the data together while ignoring the random noise that happened to be recorded alongside it.
This simulation lets you play this balance manually. A flat line (degree 1) assumes the whole world is linear. It doesn't capture the subtle pattern, so it has high bias—it is underfitting.
If you push the complexity up, you give the model enough flexibility to hit every single point. It drives the training error to near zero. But these models are brittle—a tiny change in the data makes the curve wildly different. This is high variance—overfitting.
Try pushing the degree high, then hit Resample Data. Look at how the faint ghost copies vary wildly, while a simple model stays much more consistent.
Reference
- Bias
- Error from erroneous assumptions (e.g. assuming linear when underlying function is curved)
- Variance
- Error from sensitivity to small fluctuations in the training set
- Overfitting
- Modeling noise instead of the underlying signal, characterized by low train error and high val error
- Underfitting
- Modeling neither noise nor the signal, characterized by high train and high val error
Break it on purpose
Drop the sample size to 8 and push the degree to 9. The curve now runs through almost every training point, and its peak swings hundreds of times further than the data itself ever does — validation error climbs off the chart while training error sits near zero. Then raise the sample size back up and watch the same degree-9 model calm down: capacity is only dangerous relative to how much data you have.