Ridge & Lasso Regularisation

When linear models fit noisy or collinear datasets where feature count approaches sample size ($p \approx N$), the empirical covariance matrix $X^T X$ becomes ill-conditioned. Unconstrained Ordinary Least Squares (OLS) produces wildly exploding, opposing weights ($w_1 = +42.0, w_2 = -39.0$) that cancel each other out to memorize idiosyncratic training noise. While training MSE reaches zero, test error explodes to $14.5$ due to severe model variance.

Charge Rent on Size

Regularization remedies coefficient explosion by penalizing weight magnitude directly inside the objective function:

$\mathcal{L}(\mathbf{w}) = \text{MSE}(\mathbf{w}) + \lambda \cdot \mathcal{R}(\mathbf{w})$

The total loss balances empirical fit against model complexity. For instance, with training MSE $= 1.8$, tuning parameter $\lambda = 0.5$, and weight size metric $\mathcal{R}(\mathbf{w}) = 1.8$, the regularizer charges a complexity rent of $0.5 \times 1.8 = 0.9$, bringing the total loss bill to $2.7$. Tuning $\lambda$ sweeps the Pareto frontier from pure unconstrained OLS ($\lambda = 0$) to total parameter extinction ($\lambda \to \infty$).

Ridge Shrinks Gently

Ridge regression imposes an $L_2$ Euclidean penalty $\mathcal{R}_{\text{Ridge}}(\mathbf{w}) = \sum_{j=1}^p w_j^2$. In the orthonormal setting, the closed-form analytical solution scales unpenalized OLS coefficients by a smooth shrinkage factor:

$\hat{w}_j^{\text{Ridge}} = \frac{\hat{w}_j^{\text{OLS}}}{1 + \lambda}$

With baseline $\hat{w}^{\text{OLS}} = 4.0$ and $\lambda = 0.5$, the weight shrinks by $33\%$ to $\frac{4.0}{1.5} \approx 2.67$. Geometrically, the $L_2$ constraint boundary forms a smooth circular sphere ($\|\mathbf{w}\|_2^2 \le t$). Elliptical loss contours touch this smooth arc off-axis, pulling weights continuously toward the origin without ever setting them to exact zero.

Lasso Snaps Clean

Lasso (Least Absolute Shrinkage and Selection Operator) applies an $L_1$ Manhattan penalty $\mathcal{R}_{\text{Lasso}}(\mathbf{w}) = \sum_{j=1}^p |w_j|$, governed by the soft-thresholding operator:

$\hat{w}_j^{\text{Lasso}} = \text{sign}(\hat{w}_j) \max(0, |\hat{w}_j| - \lambda)$

With $\lambda = 0.5$, a strong predictor with $\hat{w} = 3.5$ shrinks by $0.5$ to $3.0$ and survives. However, a weak noisy attribute with $\hat{w} = 0.4$ falls below the threshold ($|0.4| < 0.5$) and snaps strictly to $0.0$. Geometrically, the $L_1$ constraint forms a sharp diamond with pointed corners on the coordinate axes; quadratic loss contours frequently intersect these corners, driving coefficients to absolute zero and performing automated sparse feature selection.

Sparser Tests Better

Evaluating generalization performance on 12 held-out test points proves the statistical dividend of sparsity. A dense unregularized model retaining all 10 candidate features captures spurious noise correlations, failing on 5 of 12 test instances (error rate $0.42$). The sparse Lasso model, which zeroes out 7 irrelevant noise dimensions and retains only the 3 true signal predictors, misses only 2 of 12 test instances (error rate $0.17$), achieving a $60\%$ reduction in test error.

Match Penalty to Field

Selecting between penalties depends on the underlying data-generating distribution:

• Ridge ($L_2$): Retains all predictors, shrinking coefficients smoothly. Optimal for dense regimes where many predictors each exert small, distributed effects (e.g., climate metrics, image pixels).
• Lasso ($L_1$): Enforces true parameter sparsity. Optimal when the true signal is sparse, concentrated in a few dominant features among dozens of uninformative measurements.
• ElasticNet: Blends $L_1$ and $L_2$ penalties ($\lambda_1 \|\mathbf{w}\|_1 + \lambda_2 \|\mathbf{w}\|_2^2$), grouping correlated variables together while maintaining sparse feature selection.

Where It Breaks

Lasso encounters a severe failure mode when handling highly collinear features. Consider two identical predictors, such as height measured in centimeters ($x_1$) and height measured in inches ($x_2$). Because the $L_1$ diamond has flat planar facets between corners, the optimization problem becomes non-strictly convex: Lasso arbitrarily selects one twin at full weight ($w_1 = 1.0$) and zeroes out the other ($w_2 = 0.0$). Minor data perturbations cause the selected feature to flip randomly across runs, destroying interpretability. In such scenarios, Ridge or ElasticNet should be preferred, as their strictly convex curvature divides weight evenly across correlated twins ($w_1 = 0.5, w_2 = 0.5$).

The Quick Version

• Exploding weights: Unconstrained OLS on noisy or collinear data drives weights to massive cancelling values, causing extreme test variance.
• Size penalty: Regularization minimizes $\text{MSE} + \lambda \mathcal{R}(\mathbf{w})$, charging a complexity rent to constrain parameter growth.
• Ridge ($L_2$): Smooth circular constraint shrinks weights by $\frac{w}{1+\lambda}$ ($4.0 \to 2.67$), keeping all features active.
• Lasso ($L_1$): Sharp diamond constraint uses soft-thresholding, shrinking $3.5$ to $3.0$ while snapping $0.4$ strictly to $0.0$.
• Sparsity dividend: Dropping 7 noisy dimensions cuts held-out test error from $0.42$ (5 misses) to $0.17$ (2 misses).
• Taxonomy: Use Ridge for dense, distributed effects; use Lasso for sparse feature selection; use ElasticNet for correlated groups.
• Collinear twins limit: Lasso randomly breaks ties between identical predictors, causing unstable feature selection that Ridge resolves.