Edge of Stability

Why Does This Exist?

Classical optimization says gradient descent with step size $\eta$ diverges when the sharpness (largest Hessian eigenvalue $\lambda_{\max}$) exceeds $2/\eta$. Yet practitioners train with large steps, watch the loss wobble, and still converge. Cohen and coauthors (2021) named what happens: progressive sharpening lifts curvature until it hits the ceiling, then training runs at the edge of stability, hovering just above $2/\eta$ with non-monotone but decreasing loss.

This matters because the edge regime is where good solutions live. Small steps that satisfy the classical limit converge monotonically to the nearest minimum, often a sharp one. Large steps that ride the edge oscillate, escape sharp basins, and settle flatter. This page quantifies the ceiling, walks through the two phases, and states the practical readout. Step-size schedules are covered under learning rate schedules; the basin shapes are on the loss geometry page.

The one idea you need first

On a quadratic with curvature $\lambda$, one gradient step multiplies the distance to the minimum by $|1 - \eta \lambda|$. Below $2/\eta$ the factor is under one (converge); above it, over one (diverge, oscillating in sign). Neural nets are not quadratic, but their local sharpness plays the same role step by step.

Think of It Like This

How It Actually Works

Phase 1: progressive sharpening. From initialization, $\lambda_{\max}$ climbs steadily while the loss falls smoothly. The network enters sharper territory as features grow: larger weights mean larger curvature. This continues until $\lambda_{\max}$ reaches $2/\eta$.

Phase 2: edge of stability. Sharpness hovers at or just above $2/\eta$. The loss stops decreasing monotonically and oscillates with period two (up one step, down the next), while its running average keeps falling over tens of steps. The oscillation itself regulates sharpness: overshooting flattens the local region, undershooting sharpens it, forming a feedback loop around the ceiling.

Worked example

Quadratic $L(w) = \frac{1}{2}\lambda w^2$ with $\lambda = 3$ and $\eta = 1$. The ceiling is $2/1 = 2$, and $\lambda = 3 > 2$, so classical theory predicts divergence: factor $|1 - 3| = 2$, doubling each step with sign flips. Start $w_0 = 1$: $w_1 = 1 - 3 = -2$, $w_2 = -2 + 6 = 4$. Diverges, as advertised. Now the neural twist: suppose sharpness adapts, falling to $\lambda = 1.9$ after an overshoot. Then factor $|1 - 1.9| = 0.9$ and steps contract again. Real training alternates between these two local pictures, which is why the loss wobbles around a downward trend instead of exploding.

A realistic scale: $\eta = 0.1$ gives ceiling $20$. Cohen's experiments show sharpness climbing from about $5$ to $20$ over early training, then hovering at $20$ to $22$ for the rest, with loss oscillating visibly per step but falling per epoch.

Code

1def gd_path(lam: float, eta: float, w0: float, steps: int) -> list[float]:2    w, path = w0, [w0]3    for _ in range(steps):4        w = w - eta * lam * w  # gradient of 0.5 * lam * w^25        path.append(round(w, 3))6    return path7
8
9print(gd_path(1.5, 1.0, 1.0))  # -> stable: 1.0, -0.5, 0.25, ... shrinks10print(gd_path(3.0, 1.0, 1.0))  # -> unstable: 1.0, -2.0, 4.0, ... doubles

Curvature $1.5$ sits below the ceiling $2$ and converges; $3.0$ sits above and diverges on a fixed quadratic. Nets survive above only because curvature adapts.

Watch Out For

The Quick Version

• Gradient descent is classically stable only while sharpness stays below $2/\eta$.
• Real training sharpens progressively until it hits that ceiling, then hovers at it.
• At the edge the loss oscillates per step but trends down per epoch.
• The oscillation regulates itself: overshoots flatten, undershoots sharpen.
• With $\eta = 0.1$, sharpness climbs from about $5$ to the ceiling $20$ and stays there.
• Judge edge runs by the epoch average, and measure sharpness on large batches.

What to Read Next

• Gradient Descent Variants
• Gradients
• Convexity and Loss Landscapes
• Feature Learning vs Lazy Training