L1 & L2 Regularization Derivations

Regularization penalizes large model weights to prevent overfitting. L1 encourages weights to become exactly zero, while L2 shrinks them evenly.

Geometric views of L1 diamond constraints versus L2 circular constraints.

Why Does This Exist?

When training a machine learning model, optimizing only for the lowest error on the training set often leads to overfitting—the model learns the noise. Regularization adds a mathematical penalty for complexity, forcing the model to learn smoother, more generalizable patterns.

Think of It Like This

Imagine packing for a flight with a strict weight limit. L2 regularization is like imposing a tax on every item based on its size, so you pack smaller versions of everything. L1 regularization is a rule that says you can only bring exactly 5 items total; it forces you to completely throw out the things you don't strictly need, leaving you with a sparse suitcase.

How It Actually Works

Regularization is added directly to the loss function $J(\theta)$ :

L2 Regularization (Ridge): The penalty is the squared magnitude of the weights: $J_{reg} = J(\theta) + \lambda \sum \theta_i^2$ . When we take the derivative for gradient descent, the penalty term becomes $2\lambda \theta_i$ . This means the weight decays by a fraction of its current size at every step, shrinking exponentially but rarely reaching exactly zero.
L1 Regularization (Lasso): The penalty is the absolute magnitude of the weights: $J_{reg} = J(\theta) + \lambda \sum |\theta_i|$ . The derivative of the penalty is $\lambda \cdot \text{sign}(\theta_i)$ . This subtracts a constant amount from the weight at every step, which eventually drives less important weights to exactly zero, resulting in a sparse model.
Geometric Derivation: If framed as constrained optimization, L2 restricts weights to lie within a circle/sphere, while L1 restricts them to a diamond. The loss contours are much more likely to hit the sharp corners of the L1 diamond, snapping one of the weights to exactly zero.

Code

1import numpy as np2
3def l1_l2_gradients(weights: np.ndarray, lambda_val: float) -> tuple[np.ndarray, np.ndarray]:4    # Compute the gradient penalties for L1 and L25    l1_grad = lambda_val * np.sign(weights)6    l2_grad = 2 * lambda_val * weights7    return l1_grad, l2_grad8
9w = np.array([0.5, -2.0, 0.0])10l1_g, l2_g = l1_l2_gradients(w, 0.1)11print(f"L1 Gradient Penalty: {l1_g}")12# -> [ 0.1 -0.1  0. ]13print(f"L2 Gradient Penalty: {l2_g}")14# -> [ 0.1 -0.4  0. ]

Watch Out For

Scale imbalance: Because the penalty depends directly on the magnitude of the weights, features with vastly different scales will be penalized unevenly. You must normalize your input data before applying L1 or L2 regularization.

The Quick Version

L2 (Ridge) penalizes squared weights, causing all weights to shrink proportionally.
L1 (Lasso) penalizes absolute weights, causing less important weights to drop to exactly zero.
L1 performs feature selection inherently; L2 distributes importance across correlated features.
Both are controlled by a hyperparameter $\lambda$ that determines the strength of the penalty.