Elastic Net

One penalty picks a shortlist and a second penalty makes correlated features share weight. You keep the shortlist without letting noise decide which twin survives.

Why Does This Exist?

Picture seventy or so cell lines, thousands of gene expression columns each, and one number per line saying how hard a drug hits it. You want the genes that predict the response, because that shortlist is what the next experiment gets built around.

Lasso looks like the answer. Its penalty on total absolute coefficient size drives most coefficients to exactly zero and hands back a shortlist instead of thousands of numbers.

Then you run it on two resamples of the same rows and get two different shortlists with identical held out error. That is the wall, and it is structural. Genes arrive in co regulated clusters that rise and fall together, and lasso has no reason to prefer one member over another, so noise breaks the tie. The shortlist changes while the error stays put, which means the list was never a finding.

This page covers the fix: add a little squared penalty so the tie breaks toward sharing instead of toward extremes. The pure picking behavior lives in lasso, the pure sharing behavior lives in ridge, and the correlation pattern that causes the trouble lives in multicollinearity.

Think of It Like This

A courier with two price sheets

You must split two kilos across two identical parcels.

The first price sheet charges a flat rate per kilo. Two kilos in one parcel costs exactly what one kilo in each costs, so the sheet holds no opinion about the split. When price is indifferent something else decides, and one parcel goes out full while the other goes out empty.

The second sheet charges per kilo squared. One and one costs 1 plus 1 is 2, while two and nothing costs 4 plus 0 is 4. Even splits are strictly cheaper, so this sheet has a firm view. But it never quite empties a parcel, because the last few grams cost almost nothing to keep.

Elastic net staples the sheets together. The flat rate empties parcels with nothing worth sending, and the surcharge divides whatever is worth sending evenly instead of dumping it in one box.

Where it stops: parcels are never negative and freight is never symmetric, while penalties charge $w$ and $-w$ alike and coefficients can sit on either side of zero. The sharing logic carries over, the sign symmetry does not.

How It Actually Works

Both penalties, with one dial for strength and one for the blend:

\sum_{i=1}^{n}\big(y_i - x_i^\top w\big)^2 + \lambda\Big(\rho \lVert w \rVert_1 + \tfrac{1-\rho}{2}\lVert w \rVert_2^2\Big)

Here $w$ is the vector of coefficients, meaning one weight per feature. $\lVert w \rVert_1$ is the sum of absolute values, meaning the flat per kilo sheet. $\lVert w \rVert_2^2$ is the sum of squares, meaning the squared sheet. $\lambda$ sets total strength, and $\rho$ is the mixing ratio from $\rho = 1$ for pure lasso down to $\rho = 0$ for pure ridge. Two dials instead of one.

Worked example: two twins that must sum to 2

Take two features that move together perfectly, and suppose the data wants their coefficients to add to 2. Every split fits equally well: $(2, 0)$ , $(1, 1)$ , $(0.5, 1.5)$ . Identical predictions, so only the penalty decides.

Price the extreme split $(2, 0)$ and the even split $(1, 1)$ . The flat sheet charges $\lvert w_1 \rvert + \lvert w_2 \rvert$ , which is 2 for both. Flat across the whole segment, no preference, so the winner is settled by noise and lands at an end. The squared sheet charges $w_1^2 + w_2^2$ , which is $4 + 0 = 4$ for the extreme split and $1 + 1 = 2$ for the even one. A parabola along the segment whose bottom is the even split, and a well rather than a knife edge, so a resample barely moves the answer.

Now blend them at strength $\lambda = 1$ and ratio $\rho = 0.5$ :

0.5 \times 2 + 0.25 \times 2 = 1.5 \quad \text{for } (1, 1)

0.5 \times 2 + 0.25 \times 4 = 2.0 \quad \text{for } (2, 0)

Even a half mix tips the flat segment into a bowl. The even split wins by half a point of penalty, correlated features shrink toward each other, and they enter and leave the model together. The corners survive the mix, so features with nothing to contribute still land on exactly zero. That is the diagram above: corners intact, edges pushed out, 1.5 against 2.0.

The cap lasso cannot lift

With $n$ rows, lasso returns at most $n$ nonzero coefficients. Structural, not a tuning artifact. If the pathway holds hundreds of genes and you have seventy rows, lasso cannot say so.

The squared term removes that ceiling by making the objective strictly convex in every direction, so there is one unique solution with no reason to stop at $n$ . Wide data is where this pays: gene expression, text n grams that co occur, one hot columns off a high cardinality category. There, correlated clusters are the shape of the data, not the exception.

When not to bother

If your columns are close to independent, stay on pure lasso. Nothing to group, and you keep the sparser model with one dial instead of two. If you want no shortlist at all, use ridge: it ties elastic net on held out error more often than people expect, on one dial. Elastic net earns its second dial in exactly one situation, which is a sparse model plus correlated clusters. Common enough to matter, rare enough to check.

Code

The point is the pricing: rebuild the worked example and confirm the tie at L1, the 4 against 2 at L2, and the 1.5 against 2.0 at the half mix.

1def l1(w: list[float]) -> float:2    return sum(abs(v) for v in w)3
4
5def l2_sq(w: list[float]) -> float:6    return sum(v * v for v in w)7
8
9def elastic(w: list[float], lam: float, rho: float) -> float:10    return lam * (rho * l1(w) + (1.0 - rho) / 2.0 * l2_sq(w))11
12
13split: list[float] = [1.0, 1.0]14extreme: list[float] = [2.0, 0.0]15print(l1(split), l1(extreme))  # -> 2.0 2.016print(l2_sq(split), l2_sq(extreme))  # -> 2.0 4.017print(elastic(split, 1.0, 0.5), elastic(extreme, 1.0, 0.5))  # -> 1.5 2.0

L1 ties at 2.0, which is why lasso lets noise pick. L2 separates 2.0 from 4.0, and the half mix separates 1.5 from 2.0 while still clipping true zeros. Try $\rho = 0.9$ and watch the gap shrink toward zero: near pure lasso the bowl flattens back into a knife edge.

Watch Out For

A grid too coarse to separate the two dials, reported as a finding

The symptom: three values of $\rho$ , five of $\lambda$ , a winner at $\rho = 0.5$ , and a writeup claiming the model needs a balanced mix. Refit on a finer grid and $\rho = 0.9$ wins by less than the fold to fold spread. The ratio never meant anything. It was the argmax of noise.

Symptom in numbers: the top few settings sit inside one standard error of each other. Fix: search $\lambda$ on a log scale, weight the $\rho$ grid toward the lasso end at 0.5, 0.7, 0.9, 0.95, 0.99, 1, and report the fold spread beside the winner. When the leaders tie, pick the sparsest. Cross-validation schemes covers getting that spread honestly.

Adding a second penalty when the columns were never scaled

Expression counts in the thousands beside a ratio between 0 and 1, no standardization, and a penalty that charges the ratio coefficient thousands of times more for the same effect. Selection looks unstable, so you add a second dial, and it stays unstable. Both penalties charge by magnitude, and magnitude is set by whoever chose the units.

Symptom: coefficients track column scales more than column signal. Fix: standardize every column first, inside each fold so held out statistics never leak into training, then refit pure lasso and see whether the instability survived. Often it has not. Feature scaling has the mechanics.

The Quick Version

The penalty is $\lambda(\rho \lVert w \rVert_1 + \tfrac{1-\rho}{2}\lVert w \rVert_2^2)$ : one dial for strength, one for the blend, with $\rho = 1$ giving lasso and $\rho = 0$ giving ridge.
Splits of 2 as (2,0) and (1,1) tie at L1 cost 2, while L2 prices them 4 and 2, so the blend at mix 0.5 scores them 2.0 and 1.5.
Correlated features therefore enter and leave together, and the selected set survives a resample instead of trading places.
Axis corners survive the mix, so coefficients still reach exactly zero. Sparsity is softened, not traded away.
No cap on selected features, unlike lasso with its ceiling at $n$ rows, which is what makes the mix usable when features outnumber rows.
The price is a second hyperparameter. Skip it when columns are near independent or when no shortlist is needed.