Ridge & Lasso Regularisation
How L1 and L2 penalties stop models from memorising noise, and why their different geometries determine whether features get shrunk or discarded.
Given enough features, a model can perfectly memorise a training set. It does this by assigning wildly large coefficients (weights) that mathematically cancel each other out to fit random noise. Regularisation fixes this by penalising the model for using large weights.
Ridge: Uniform Shrinkage
Ridge (L2) regularisation adds a penalty based on the square of the weights. This pushes every coefficient closer to zero. They shrink, meaning the model relies less on any single feature, but the coefficients rarely reach exactly zero.
Lasso: Feature Selection
Lasso (L1) regularisation adds a penalty based on the absolute value of the weights. This behaves differently: it pushes less important coefficients all the way to absolute zero. Lasso doesn't just shrink features; it deletes them, performing automated feature selection.
The Geometric Reason
The difference in behaviour comes from geometry. The L2 penalty forms a sphere, so the optimal trade-off point is almost always somewhere on the curve. The L1 penalty forms a diamond. Its sharp corners sit exactly on the axes, so the optimal trade-off point hits the corner, forcing one of the weights to be zero.
Where It Breaks
If your dataset contains two highly correlated features (e.g., height in inches and height in cm), Lasso will arbitrarily pick one to keep and completely zero out the other. If you need both, Lasso fails. Elastic Net fixes this by applying both L1 and L2 penalties at once.
The Quick Version
- The problem: Unconstrained models use huge weights to overfit.
- Ridge (L2): Shrinks all weights smoothly. Good for avoiding overfitting.
- Lasso (L1): Shrinks some weights to zero. Good for feature selection.
- Geometry: L1's diamond shape guarantees intersections on the axes.
- Failure: Lasso arbitrarily drops one of a pair of correlated features.