Saddle Point Escape

Saddle points stall gradient descent with near-zero slopes, so escape methods add noise or curvature steps that find the few downhill exits.

A small random kick plus a curvature step exits a saddle along its negative eigendirection

Why Does This Exist?

Near a saddle, the gradient shrinks toward zero while a downhill exit stays open along the negative-curvature direction. Plain gradient descent slows to a crawl exactly there: steps scale with the gradient, so the smaller the slope, the smaller the step, right when you need a decisive move.

Training runs stall for hundreds of steps on these passes. Escape methods break the deadlock with two ingredients: noise that knocks iterates off the flat ridge, and curvature exploitation that aims down the exit once it appears. Neighbours: the terrain map (non-convex landscapes) and the convergence rates (non-convex SGD).

The one idea you need first

A strict saddle is a critical point whose Hessian has at least one eigenvalue below some negative threshold $-\gamma$ . That gap matters: it guarantees a real downhill direction rather than an arbitrarily flat one. Gradient descent theory in stochastic gradient descent variants shows why plain steps slow down there.

Think of It Like This

A ball balanced on a horse saddle

Set a ball exactly in the middle of a horse saddle. Along the horse it rolls down to the ground. Across the horse it rolls back to the middle. Balanced perfectly, it sits still, and a feather touch decides its fate.

Gradient descent is that feather touch missing: with no noise it creeps along the flat and barely moves. A small random tap rolls it slightly sideways, and then gravity (the negative curvature) takes over and pulls it down the exit fast.

Where the analogy stops: real saddles have thousands of stable directions holding the iterate in place, so the tap must be repeated, not once.

How It Actually Works

The strict-saddle program has three moves:

Detect the stall. Gradient norm below $\epsilon$ with iterations still at high loss.
Perturb. Add uniform noise of radius $r$ (typical theory choice scales with $\epsilon$ ) to leave the thin stuck region.
Descend or curve. Run gradient steps, which now amplify the unstable direction exponentially, or take an explicit negative-curvature step $x \to x - \alpha v$ where $v$ is the bottom eigenvector.

Why it works: near a strict saddle the dynamics separate. Stable directions contract and the unstable one expands like $(1 + \eta\gamma)^t$ . After $O(\log(1/\epsilon))$ steps the exit component dominates and the loss drops by a fixed amount. Repeating gives a polynomial-time guarantee to an approximate second-order stationary point.

Worked example

Take $f(x, y) = x^2 - y^2$ starting at $(0.01, 0.0)$ with step $\eta = 0.1$ :

Gradient: $[0.02, 0]$ . Plain step gives $(0.008, 0)$ . Still stuck near the ridge.
Add noise $(0, 0.05)$ : position becomes $(0.008, 0.05)$ .
Next gradient: $[0.016, -0.1]$ . Update: $x \to 0.0064$ , $y \to 0.06$ . The $y$ component grows 20 percent per step while $x$ shrinks.
After 10 such steps $y \approx 0.05 \times 1.2^{10} \approx 0.31$ and loss falls from $0$ to about $-0.096$ . The exit amplifies itself once noise reveals it.

Code

1import numpy as np2
3
4def grad(w: np.ndarray) -> np.ndarray:5    return np.array([2.0 * w[0], -2.0 * w[1]])  # grad of x^2 - y^26
7
8def perturbed_descent(w0: np.ndarray, steps: int = 30) -> np.ndarray:9    w = w0.copy()10    rng = np.random.default_rng(0)11    for _ in range(steps):12        if np.linalg.norm(grad(w)) < 0.05:  # stalled: kick off the ridge13            w = w + rng.uniform(-0.05, 0.05, size=2)14        w = w - 0.1 * grad(w)15    return w16
17
18print(perturbed_descent(np.array([0.01, 0.0])))  # -> [~0.0  ~0.3], escaped along y

Watch Out For

Adding noise so large it erases progress

Big kicks exit saddles fast but also eject the iterate from good valleys. Loss spikes back up and training restarts from scratch every few hundred steps.

Keep the perturbation radius tied to the target accuracy (theory uses $r \approx \epsilon$ scale) and perturb only when the gradient norm is small, not on every step.

Mistaking a plateau for a strict saddle

Perturbation guarantees need a real negative eigenvalue. On a flat plateau every eigenvalue sits near zero, so noise just diffuses and no direction amplifies.

If repeated kicks produce no loss drop, you face a plateau or valley, not a saddle. Switch to larger steps or momentum instead of more noise.

The Quick Version

Strict saddles have a downhill exit marked by a negative Hessian eigenvalue below a gap.
Plain gradient steps shrink with the gradient and stall exactly on the pass.
Small noise reveals the exit, then dynamics amplify it exponentially fast.
Perturb only when stalled and keep the radius near the accuracy scale.
Plateaus have no negative eigenvalue, so noise alone cannot fix them.