Ridge Regression

Add a penalty on coefficient size and the bias variance tradeoff becomes a dial. You accept a little bias on purpose to buy a much bigger drop in variance.

Raising the penalty pulls the coefficient from 2.036 at zero penalty to 1.9 at lambda 1 and 1.188 at lambda 10, shrinking smoothly without ever landing on zero

Why Does This Exist?

Thirty wheat fields, forty sensor readings each, one yield number per field. Two soil moisture probes sit three meters apart and agree all season within rounding error. Plain least squares threads the thirty training points a fraction tighter by giving those two near identical columns huge cancelling weights, one large positive and one large negative.

Then someone recalibrates a probe. The cancellation breaks and held out error jumps. Nothing about wheat changed. The objective simply had no opinion about weight size, so it spent enormous weights to buy a tiny training gain. The full least squares setup is in ordinary-least-squares. This page adds the one term that gives the objective that missing opinion, and its sibling penalties are in lasso-regression and elastic-net.

Think of It Like This

Springs pulling every dial toward zero

You tune forty dials on a mixing desk to match a recording, and many settings match about equally well.

Now hook a spring from each dial to its zero mark, with the spring cost growing as the square of the distance pulled. The second centimeter costs more than the first. Huge cancelling pairs stop being affordable, because one dial far positive and its twin far negative stretches two springs hard for almost no audible gain. A flat shared setting costs almost nothing in spring energy and survives a recalibrated probe.

No dial ever snaps onto zero. A spring pull weakens as the dial closes in, and at zero there is no pull left. Dials slide arbitrarily close and park there.

Where it stops: real springs push back physically while the penalty only changes the score being optimized. Turn the dial up far enough and the math still lets weights grow, so a bad enough fit can overpower the spring in a way a real spring would forbid.

How It Actually Works

Least squares chooses weights $w$ to minimize the sum of squared misses over $n$ rows. Here $y_i$ is row $i$ target, $x_i$ is its feature row, and $w$ is the coefficient vector. Ridge adds one term:

\sum_{i=1}^{n} (y_i - x_i^T w)^2 + \lambda w^T w

The added term $w^T w$ is the sum of squared coefficients, the squared L2 norm. The number $\lambda$ is the dial. At $\lambda = 0$ this is plain least squares again. As $\lambda$ grows, every coefficient gets squeezed toward zero.

The ridge on the diagonal

Setting the derivative to zero gives a closed form. With $X$ the full data table and $I$ the identity matrix:

w = (X^T X + \lambda I)^{-1} X^T y

The term $\lambda I$ adds $\lambda$ to each diagonal entry of $X^T X$ and touches nothing else. That diagonal ridge is where the name comes from. It is also the numerical fix: with more columns than rows, $X^T X$ is singular and least squares has no unique answer, while lifting every eigenvalue by exactly $\lambda$ makes the matrix invertible. Two near identical probes are the same disease in milder form, an eigenvalue near zero rather than at it.

Standardize first and spare the intercept

The penalty charges by coefficient size, and size depends on units. A column measured in large units needs only a small coefficient to move predictions, while the same signal measured in tiny units needs a huge one, and squaring magnifies that gap. So standardize every column first by subtracting its mean and dividing by its spread. Skip that and $\lambda$ penalizes your choice of units instead of your model.

Leave the intercept out of the penalty. It is the prediction when every feature sits at its average, and shrinking it toward zero asserts that an average row should predict zero. Every implementation excludes it.

Worked example

Take one slope through three points with $x = [1.0, 2.0, 3.0]$ and $y = [2.1, 3.9, 6.2]$ . The needed sums are $s_{xy} = 1.0 \times 2.1 + 2.0 \times 3.9 + 3.0 \times 6.2 = 28.5$ and $s_{xx} = 1.0 + 4.0 + 9.0 = 14.0$ . The plain estimate is $28.5 / 14.0 = 2.0357$ , which rounds to $2.036$ . Ridge divides by $s_{xx} + \lambda$ instead, so at $\lambda = 1.0$ the estimate is $28.5 / 15.0 = 1.9$ , and at $\lambda = 10.0$ it is $28.5 / 24.0 = 1.1875$ , which rounds to $1.188$ . Each step shrinks smoothly. None of them lands on zero, because the squared penalty pull fades as the coefficient does. The diagram above plots exactly these three points.

Code

The snippet below recomputes the worked example with only the standard library. It shows the shrinkage path directly: larger $\lambda$ means a smaller slope, every time.

1xs = [1.0, 2.0, 3.0]2ys = [2.1, 3.9, 6.2]3
4sxy = sum(x * y for x, y in zip(xs, ys))5sxx = sum(x * x for x in xs)6
7print(round(sxy / sxx, 3))  # -> 2.0368print(round(sxy / (sxx + 1.0), 3))  # -> 1.99print(round(sxy / (sxx + 10.0), 3))  # -> 1.188

The three numbers match the hand computation line for line. Picking between them is a held out error decision, since no value of $\lambda$ is derivable from the fitted data alone.

Watch Out For

Fitting on raw units then trusting the chosen lambda

The symptom is a chosen $\lambda$ that swings by orders of magnitude when one column changes units, with the surviving coefficients reshuffling alongside it. The soil never changed. The penalty measured your ruler.

Standardize inside each cross validation fold, not once across the whole table first. Computing means and spreads over all rows before splitting leaks held out statistics into training. A pipeline that scales after the split avoids that quiet data leakage.

Reporting the tuned score as the honest score

The symptom is a cross validated error that looks strong in review and weak in production. You searched many values of $\lambda$ , kept the winner, and reported the winner score. Winners win partly on merit and partly on lucky folds.

Split off a test set before tuning and score it once at the end. When the data are too small to spare rows, use nested validation with an inner loop that picks $\lambda$ and an outer loop that scores the whole pick then fit procedure.

The Quick Version

Ridge adds $\lambda w^T w$ to squared error, which is one term plus one number to tune.
The closed form gains $\lambda I$ on the diagonal, lifting every eigenvalue by $\lambda$ so the solve works even with more features than rows.
Coefficients shrink smoothly toward zero and never arrive, because a squared penalty pull vanishes as the coefficient does.
Correlated features share the load instead of fighting over it with cancelling extremes.
Standardize features first and never penalize the intercept, or $\lambda$ punishes units instead of complexity.
The bias is bought on purpose. Only held out rows untouched by tuning can judge whether the trade paid.