Overfitting vs Underfitting
How to diagnose if your model is memorizing the data or failing to learn anything at all by looking at learning curves.
Verdict: Plot the training and validation loss. If both are high, you are underfitting. If they diverge (train drops, validation rises), you are overfitting.
The Short Answer
The ultimate goal of machine learning is generalization: building a model that performs well on data it has never seen before.
- Underfitting happens when a model is too simple to capture the underlying patterns in the data. It performs poorly on both the training data and the unseen test data.
- Overfitting happens when a model is too complex. It doesn't just learn the underlying patterns; it memorizes the random noise and specific quirks of the training data. It performs flawlessly on the training data, but horribly on unseen test data.
Where They Differ
| Feature | Underfitting | Overfitting |
|---|---|---|
| Model Complexity | Too low (e.g., linear model for curved data). | Too high (e.g., massive deep neural network on tiny data). |
| Training Error | High. | Near Zero. |
| Validation Error | High. | High. |
| Analogy | Failing a math test because you didn't study at all. | Failing a math test because you memorized the practice test answers without learning the concepts. |
Diagnosis from Curves
You cannot diagnose these problems by just looking at the final accuracy number. You must plot the Learning Curves: the loss (error) on both the Training set and the Validation set over time (epochs).
The Underfitting Curve
The training loss drops a tiny bit and then plateaus at a high value. The validation loss tracks right alongside it. The model has hit a wall and cannot learn any more.
The Overfitting Curve (The U-Shape)
Initially, both the training and validation loss drop together. But eventually, a split happens. The training loss keeps going down towards zero, but the validation loss bottoms out and actually starts going back up. This is the exact moment the model stopped learning general patterns and started memorizing noise.
How to Fix Underfitting
If your curves show underfitting, your model is choked. It needs more power.
- Increase Model Complexity: Add more layers to your neural network, or use a non-linear model instead of a linear one.
- Add more features: Provide the model with more columns of data (feature engineering) so it has more information to learn from.
- Train longer: Sometimes, you just stopped the training process too early.
How to Fix Overfitting
If your curves show the classic U-shape, your model is too powerful for the amount of data you have.
- Early Stopping: The easiest fix. Look at the curve, find the exact epoch where the validation loss hit its lowest point before rising, and use the model from that epoch.
- Get more data: A complex model is harder to overfit if it is forced to look at millions of varied examples.
- Regularization: Add techniques like L1/L2 penalties, Dropout layers, or Data Augmentation to artificially handicap the model's ability to memorize.
What People Get Wrong
Chasing 100% Training Accuracy
Beginners often celebrate when their model hits 99.9% accuracy on the training set. In almost all real-world scenarios, a model with 99.9% training accuracy is severely overfitting. A healthy, generalized model usually has a small gap between train and validation performance.