Lasso Regression

Penalise the absolute size of each coefficient and the penalty keeps full force at zero, so weak coefficients land exactly on it. Shrinkage becomes selection.

The error contour touches the L1 diamond at a corner where one coefficient is exactly zero, while the L2 circle is touched mid-arc with both nonzero

Why Does This Exist?

Sixty delivery vans report nine hundred telemetry channels each, and one number per van: millimetres of brake pad lost over 40,000 km. The fleet manager wants a shortlist, because the next fleet ships with twelve sensors and someone must say which twelve.

Ridge regression cannot answer that. It returns nine hundred small nonzero coefficients, and 0.0004 is not a statement that a channel does not matter.

One idea first. Least squares fits one coefficient per channel, meaning how much predicted wear moves per unit of that reading, to minimise squared misses. A penalty adds a second charge on coefficient size. Ridge charges squared size. Lasso charges absolute size.

One swapped exponent, and the difference is shrinking versus selecting.

This page covers the selecting penalty. The squared twin it is always compared against is ridge regression, the blend of the two is elastic net, and the overfitting problem both address is overfitting and underfitting.

Think of It Like This

Friction floor versus spring tether

Picture one coefficient as a crate on a chalk line, with the evidence in the data pushing it off the line and the penalty resisting.

Tether the crate to the line with a spring and resistance dies near the line itself. On the line the spring is slack, so the faintest push nudges the crate off. That is a squared penalty, and it is why ridge never returns zero.

Now untie the spring and stand the crate on rough concrete. Friction pushes back with the same force at every distance, including at zero. A push weaker than friction moves nothing. A stronger push slides the crate but stops short by exactly friction worth.

The crate is the coefficient, the push is the evidence, the friction is lambda, and stopping at zero is sparsity. The arithmetic matches part for part: evidence weaker than the penalty stays at exactly zero, stronger evidence shifts inward by the penalty.

Where it stops: crates sit still on their own, while coefficients are recomputed jointly, so with two identical channels the survivor is a coin flip rather than a verdict about which sensor is better.

How It Actually Works

Same squared error as always, one term swapped in:

\sum_{i=1}^{n}\big(y_i - x_i^\top w\big)^2 + \lambda \lVert w \rVert_1

Here $y_i$ is van $i$ measured wear, $x_i$ its row of readings, $w$ the coefficients being solved for, $\lVert w \rVert_1$ the L1 norm that sums absolute values without squaring, and $\lambda$ the penalty strength.

Why corners do the selecting

Fix the total absolute size at some budget $t$ and minimise error inside it. With two coefficients the budget region $\lvert w_1 \rvert + \lvert w_2 \rvert \le t$ is a diamond with corners on the axes, and sitting on an axis means one coefficient is exactly zero.

Squared error contours are ellipses centred on the unpenalised fit. Grow one until it first touches the diamond, as the diagram draws. A corner catches touches from a whole wedge of directions while an edge point catches only one, so the first touch lands on a corner most of the time. Zeros are what cornered regions do, not luck.

Redraw the budget with squared size and the region is a circle with no corners, touched mid-arc with both coefficients nonzero. Shrinking, not selecting. The side-by-side visual in ridge versus lasso regularisation draws the same pair.

Worked example: the soft threshold

One coefficient, penalty $\lambda = 1$ . The single-variable answer slides the evidence toward zero by $\lambda$ and stops at zero if it crosses:

S(3.0, 1) = 3.0 - 1 = 2.0 \qquad S(0.5, 1) = 0

Evidence 3.0 survives, shortened to 2.0. Evidence 0.5 is weaker than the friction and collapses to exactly 0. Negative evidence behaves symmetrically: $S(-2.5, 1) = -1.5$ . Friction written as arithmetic.

No closed form and a straight-line path

Ridge collapses into one matrix equation because squares differentiate. $\lvert w \rvert$ has no derivative at zero, its slope jumping from $-1$ to $+1$ , so coordinate descent sweeps one coefficient at a time with the threshold above instead. The objective stays convex, so the sweeps reach a global optimum.

Sweep $\lambda$ from huge down to nothing and each coefficient traces straight segments that kink only when a channel enters or leaves the nonzero set. One ceiling matters: with sixty vans lasso keeps at most sixty nonzero coefficients, a property of the solution rather than a tuning artefact.

Code

The point is the threshold: strong evidence shortened, weak evidence zeroed.

1def soft_thresh(evidence: float, lam: float) -> float:2    if evidence > lam:3        return evidence - lam4    if evidence < -lam:5        return evidence + lam6    return 0.07
8print(soft_thresh(3.0, 1.0))  # -> 2.09print(soft_thresh(0.5, 1.0))  # -> 0.010print(soft_thresh(-2.5, 1.0))  # -> -1.5

The printout matches the hand computation: 3.0 becomes 2.0, 0.5 becomes 0, and negative evidence mirrors the positive side. Raise lambda to 3.5 and watch 3.0 collapse to zero as well.

Watch Out For

Reading survivors as the important channels

The symptom is a slide naming twelve key signals, and a set that reshuffles on next month data. Lasso answers which small set predicts well, and with two correlated channels it keeps whichever correlated a hair better. Zero looks like a verdict but it is a tie-break.

Symptom plus fix: report correlation groups rather than lone winners, reading the shortlist as one representative per cluster. Ranking inside a cluster belongs to feature selection, not to the penalty.

Ordering twelve sensors off one fit

Nine hundred columns, sixty rows, one fit, twelve survivors, one purchase order. Refit on a resample and half the twelve change, with no warning from the single run.

Symptom: selection frequencies near 30 percent for channels the shortlist treats as certain. Fix: refit on hundreds of resamples and act on selection frequency. Ninety percent is a finding, thirty is noise wearing a coefficient, and clusters that keep trading places want elastic net.

The Quick Version

Lasso adds lambda times the L1 norm to squared error, and its pull holds constant to zero so weak coefficients arrive and stay.
The diamond budget has corners on the axes, and a growing error ellipse meets a corner from many directions but an edge from one.
Evidence 3.0 survives as 2.0 at lambda 1 while evidence 0.5 collapses to exactly 0.
Absolute value has no derivative at zero, so coordinate descent replaces the closed form while convexity keeps the global optimum.
At most n nonzero coefficients come out of n rows, and correlated features are picked between arbitrarily.
Predictions survive that arbitrariness while interpretations do not, so shortlists need resampling before spending.