Evaluation & RL
Learning Curves
How adding more training data helps a model generalize (lowering validation error) but makes the training set harder to memorize perfectly (raising training error).
Stage 1 of 3: The Small Dataset
Train error 0.00, Val error 100.00
- Train Data
Validation error exploded! The model memorized the few training points by wildly oscillating.
Check your understanding
1 questions in the bank. Each attempt draws a fresh set in a fresh order, so a second go is a real second go.
Plot train vs validation error as dataset size grows to visually diagnose underfitting and overfitting.
A Learning Curve is one of the most powerful diagnostic tools in machine learning. It plots the model's error on the training set and the validation set as a function of the amount of training data.
When you have very little data, a complex model can easily memorize every point. The training error is nearly zero, but because it has learned the noise instead of the signal, its error on unseen validation data is massive. This is Overfitting.
As you add more data, the model can no longer hit every point perfectly. The training error goes up. However, the model is forced to learn the actual underlying pattern, so the validation error goes down.
By looking at the gap between the two curves, you can diagnose what your model needs. If the gap is large, you are overfitting (add more data or reduce model capacity). If they converge but both are high, you are underfitting (increase model capacity).
Reference
- Learning Curve
- A plot showing model performance vs experience (usually training set size).
- Overfitting
- High variance. Large gap between train and val error.
- Underfitting
- High bias. Train and val error converge, but both are high.
Break it on purpose
Using a high-capacity model on very few data points causes the training error to drop to zero while validation error explodes—a classic visualization of overfitting.