Subgradients

Kinks have no single slope, but every line touching from below still works. Subgradients collect those lines so nonsmooth convex problems run.

Absolute value has slope minus 1 left and plus 1 right, with every slope between valid at the kink including flat zero

Why Does This Exist?

Lasso, SVM hinges, and ReLU all kink: no gradient at zero, yet optima sit exactly there. Demanding differentiability either smooths away the sparsity you wanted or stalls at the interesting point. Training sparse and robust models needs calculus that works at corners.

Subgradients supply it: $g$ is a subgradient at $x$ when $f(y) \\geq f(x) + g^\\top(y - x)$ everywhere, a global under-estimator. The set $\\partial f(x)$ drives optimality ( $0 \\in \\partial f(x^\\star)$ ) and the subgradient method. It extends convex sets and functions into the nonsmooth half of convex optimization.

The one idea you need first

A convex smooth function's tangent sits below the graph. At a kink, several lines do. For $f(x) = |x|$ : left slope $-1$ , right slope $+1$ , and at 0 every $g \\in [-1, 1]$ under-supports. Zero belongs, so 0 is optimal.

Think of It Like This

Propping a tent at the pole

A tent's peak is a kink: no single roof slope exists there. Any plank touching the peak from below without piercing canvas is a valid prop. Steep planks lean left or right; a flat plank balances on top.

All props certify the peak holds canvas up. Where it stops: planks are straight and global, so on non-convex tents (sagging canvas) a local prop can pierce far away and the certificate fails.

How It Actually Works

Calculus: $\\partial(\\sum f_i) = \\sum \\partial f_i$ (mild conditions), chain and max rules extend by convex hulls. Optimality $0 \\in \\partial f(x^\\star)$ is necessary and sufficient for convex $f$ . Subgradient method $x_{k+1} = x_k - \\alpha_k g_k$ with diminishing steps ( $\\alpha_k \\sim 1/\\sqrt{k}$ ) converges $O(1/\\sqrt{k})$ in function value: slow but universal. Lasso $|w|$ uses soft-thresholding: $w \\leftarrow \\mathrm{sign}(w)\\max(|w| - \\alpha\\lambda, 0)$ .

Worked example

Minimize $|w| + (w - 0.4)^2$ : smooth part pulls toward 0.4, kink pulls toward 0. Subgradient condition: $0 \\in \\partial|w| + 2(w - 0.4)$ . Try $w = 0$ : $\\partial|0| = [-1,1]$ , so need $g + 2(-0.4) = 0$ , i.e. $g = 0.8 \\in [-1,1]$ . Certified: $w^\\star = 0$ . Lasso step check from $w = 0.4$ , $\\lambda = 1$ , $\\alpha = 0.5$ : shrink $0.4 - 0.5 = -0.1 \\to$ crosses zero $\\to 0$ . Sparsity in one line.

Code

1w = 0.42alpha, lam = 0.5, 1.03w = float(__import__('math').copysign(max(abs(w) - alpha * lam, 0.0), w))4print(round(w, 2))  # -> 0.0

Shrinkage zeroes 0.4 in one proximal step. The kink eats small weights exactly, which is the sparsity mechanism.

Watch Out For

Fixed steps on the subgradient method

Constant $\\alpha$ on nonsmooth objectives oscillates around the kink forever: the method steps over zero by $\\alpha$ each time and never settles. Loss plateaus above optimum and teams blame the model.

Diminish ( $1/\\sqrt{k}$ ) or switch to proximal/bundle methods. If iterates bounce across zero with fixed amplitude, the schedule is the bug.

Picking ReLU subgradient 1 at zero everywhere

Autodiff returns 0 or 1 at ReLU zero arbitrarily per framework. A network with many dead units plus subgradient 1 leaks phantom signal; with 0 it kills recovery. Either choice biases thin-margin behavior.

Treat exact-zero as measure-zero noise in training, but audit dead-unit counts. If dead units dominate, the kink convention is not your problem; initialization is.

The Quick Version

Subgradients are global linear under-estimators; the set at a kink replaces the missing gradient.
$|x|$ has $\\partial f(0) = [-1, 1]$ ; zero inside certifies optimality.
Lasso zeroes small weights by soft-thresholding in one step.
The subgradient method needs diminishing steps and runs $O(1/\\sqrt{k})$ .
Optimality $0 \\in \\partial f(x^\\star)$ is necessary and sufficient for convex $f$ .
Fixed steps oscillate; ReLU-at-zero conventions need auditing, not worship.