L1 & L2 Regularization Derivations

Why Does This Exist?

Training loss alone rewards memorisation: a model with enough weights can fit noise exactly and fail on anything new. Regularization adds a penalty for complexity, trading a slightly worse fit on the training set for weights that generalise. Both penalties below do that, yet they produce visibly different models: one sparse, one dense.

This page derives why, from the gradients. The applied side, fitting each penalty on real regressions and reading the coefficients, lives in lasso regression and ridge regression. The unifying view, that minimising $f(x) + \lambda \lVert x \rVert^2$ is the Lagrangian of minimising $f(x)$ subject to $\lVert x \rVert^2 \le c$, lives in constrained optimisation and duality, with convexity background in convex optimization.

The one idea you need first

Gradient descent repeats $w \leftarrow w - \eta \nabla L$, where $\eta$ is the step size. A penalty $P(w)$ added to the loss adds its own gradient $P'(w)$ to that update. Everything below is one question: what does $P'(w)$ look like near zero, where a weight is deciding whether to survive.

Think of It Like This

How It Actually Works

Add the penalty to the loss $J(w)$, with strength $\lambda > 0$ and weights $w_1, \ldots, w_n$. In plain words: $J$ measures fit, the penalty measures complexity, $\lambda$ sets the exchange rate.

L2 (ridge): $J_{\mathrm{reg}} = J(w) + \lambda \sum_i w_i^2$. The penalty gradient is $2\lambda w_i$, proportional to the weight itself. A gradient step multiplies the weight by $(1 - 2\eta\lambda)$: exponential shrinkage toward zero that never arrives in finite steps. Zero is a fixed point the iterates approach but do not land on.

L1 (lasso): $J_{\mathrm{reg}} = J(w) + \lambda \sum_i \lvert w_i \rvert$. Away from zero the penalty gradient is $\lambda \cdot \mathrm{sign}(w_i)$, a constant independent of size. A gradient step subtracts the same fixed amount $\eta\lambda$ every round, so small weights cross zero in finite steps. At exactly zero the absolute value has no single derivative; instead the subgradient is the whole interval $[-\lambda, \lambda]$, which contains a value that cancels the loss gradient and locks the weight at zero. That kink is the entire difference between the two penalties.

Geometry. Read as constrained optimization, L2 restricts weights to a ball and L1 to a cross-polytope, a diamond in two dimensions. Loss contours expanding from the unregularised minimum touch the ball at a generic point with all coordinates nonzero, but touch the diamond at a corner with high probability, snapping one coordinate to exactly zero. The diagram above draws both contacts.

Worked example

One weight, quadratic fit $(w - 4)^2$, so the unregularised answer is $w = 4$. Penalise it and solve in closed form.

L2 with $\lambda = 1$: minimise $(w - 4)^2 + w^2$. The derivative $2(w - 4) + 2w = 4w - 8$ vanishes at $w^{\star} = 2$. Halved, not zeroed.

L1 with strength $\alpha = 6$: minimise $(w - 4)^2 + 6\lvert w \rvert$. For $w > 0$ the derivative $2(w - 4) + 6$ vanishes at $w = 1$, which is positive and consistent, so $w^{\star} = 1$. This is soft thresholding: $w^{\star} = \mathrm{sign}(z)\max(\lvert z \rvert - \alpha/2, 0)$ for target $z = 4$.

Raise the L1 strength to $\alpha = 10$: the formula gives $\max(4 - 5, 0) = 0$, so $w^{\star} = 0$, exactly. Check the lock: at $w = 0$ the loss gradient is $2(0 - 4) = -8$ and the subgradient interval is $[-10, 10]$, which contains $+8$ to cancel it. Locked.

Contrast with strong L2, $\lambda = 5$: $w^{\star} = 4/(1 + 5) \approx 0.667$. Small, dense, nonzero. Same target, same spirit of penalty, opposite fate for the weight.

Code

The point is the check: the closed forms above, evaluated, with the dense L2 answer beside the sparse L1 ones.

1import numpy as np2
3
4def soft_threshold(z: float, alpha: float) -> float:5    return float(np.sign(z) * max(abs(z) - alpha / 2, 0.0))  # L1 closed form6
7
8print(round(soft_threshold(4.0, 6.0), 3))  # -> 1.09print(round(soft_threshold(4.0, 10.0), 3))  # -> 0.0   exactly sparse10print(round(4.0 / (1 + 5.0), 3))  # -> 0.667  L2 with penalty 5 stays dense

Strength 6 leaves weight 1, strength 10 locks it at 0, while L2 at strength 5 leaves 0.667. Sparsity against shrinkage, in three numbers.

Watch Out For

The Quick Version

• L2 adds $\lambda \sum w_i^2$ with gradient $2\lambda w_i$: proportional shrinkage by $(1 - 2\eta\lambda)$ per step, never reaching zero.
• L1 adds $\lambda \sum \lvert w_i \rvert$ with constant-magnitude steps $\lambda \cdot \mathrm{sign}(w_i)$: small weights hit zero in finite steps and lock there via the subgradient interval.
• Soft thresholding gives the 1D answer in closed form: $\mathrm{sign}(z)\max(\lvert z \rvert - \alpha/2, 0)$.
• On target 4, L1 at strength 6 gives 1 and at strength 10 gives exactly 0, while L2 at strength 5 gives 0.667.
• Geometrically the loss contours touch the L1 diamond at a corner (sparse) and the L2 ball at a generic point (dense).
• Standardize features before penalizing, and use a proximal or coordinate-descent solver when exact zeros are the goal.

What to Read Next

• Convex Optimization
• High-Dimensional Regression
• Bias-Variance Decomposition