Critical Points and Saddle Points

Flat in every direction is only a candidate. Curvature votes next: all up is a valley, all down is a hill, and a split vote is a mountain pass.

The hyperbolic contours of x squared minus y squared cross at the origin, rising to 1 along x and falling to minus 1 along y, which is the signature of a saddle

Why Does This Exist?

Training hunts minima by walking downhill until the slope vanishes. But vanishing slope is a statement about flatness, not about being at the bottom. The walk can stop at a valley floor, a hilltop, or a mountain pass, and in high dimensions the pass is by far the most common stop. Treating every flat point as a minimum turns plateaus into false convergences and wastes the run.

This page is the multivariable sequel to critical points, maxima and minima, which sorts flat points in one variable. Here the sorting uses curvature in every direction at once, read from the Hessian. The escape artistry, how noisy steps leave saddles, belongs to saddle point escape and calculus and optimization; this page teaches the identification.

The one idea you need first

A critical (stationary) point is where all partial derivatives vanish at once, so the gradient is the zero vector. That is necessary for any interior peak or valley and sufficient for nothing. The single-variable page shows the one-dimensional sorting; the Hessian below is its multi-dimensional form.

Think of It Like This

A mountain pass at night

Stand on a pass between two peaks with a valley ahead and behind. Under your boots the ground feels level in every direction, exactly as a summit or a valley floor would feel. The level feeling cannot tell the three apart.

What distinguishes them is the curvature vote. Step north and south and you climb toward the peaks; step east and west and you descend into the valleys. Up in some directions and down in others is the pass. All up is a bowl, all down is a dome. Second derivatives cast those votes, one per direction, and the Hessian collects the ballots.

The analogy stops at dimensionality. A pass has one up axis and one down axis. A network has millions of axes, and a critical point there is almost never all-up or all-down: with that many votes, a split is a near certainty, which is why high-dimensional training meets passes, not traps.

How It Actually Works

At a critical point, the local shape is governed by the Hessian matrix $H$ of second partials. Its eigenvalues are the curvatures along its principal directions:

All eigenvalues positive: the surface bowls upward in every direction. Local minimum.
All eigenvalues negative: it domes downward everywhere. Local maximum.
Mixed signs: up in some directions, down in others. Saddle point.
Some eigenvalue zero: the test is inconclusive, and higher-order terms decide. The classic case is $x^4$ , flat and positive, versus $x^3$ , flat and crossing, both with zero curvature at the origin.

In plain words: the gradient going quiet tells you to stop walking, and the Hessian tells you where you stopped. Note the asymmetry that matters for ML. A minimum needs every eigenvalue positive, while a saddle needs just one negative direction. With $n$ directions, the chance that all $n$ votes come up positive shrinks like $2^{-n}$ , so in large models essentially every critical point is a saddle, and plateaus are passes being crossed slowly.

Worked example

Take $f(x, y) = x^2 - y^2$ at the origin.

Gradient: $[2x, -2y] = [0, 0]$ . Critical point confirmed. Hessian:

H = \begin{bmatrix} 2 & 0 \\ 0 & -2 \end{bmatrix}

Eigenvalues $2$ and $-2$ , read straight off the diagonal. Mixed signs, so this is a saddle. Walk it to feel the vote: along the $x$ axis, $f(1, 0) = 1$ , uphill. Along the $y$ axis, $f(0, 1) = -1$ , downhill. Level ground at the origin, rising one way and falling the other, exactly the mountain pass.

Contrast $g(x, y) = x^2 + 3y^2$ . Same zero gradient at the origin, but its Hessian is diagonal with $2$ and $6$ , both positive: a genuine minimum. Same flatness, opposite verdict. Flatness proposes, curvature disposes.

Watch Out For

Declaring victory at a small gradient norm

A gradient norm near zero with a loss far above expectations is usually a saddle, not a minimum, and stopping there locks in a bad model. The signature is a stalled loss with a small but non-collapsing gradient. Check curvature cheaply with Hessian-vector products before accepting the point, and prefer momentum plus minibatch noise, which follow negative-curvature directions out.

Trusting the test when an eigenvalue is zero

Zero eigenvalues mean flat directions the quadratic approximation cannot see, and the point can be a minimum, a maximum, or a saddle regardless of the other signs. Ridges in overparameterized models produce exact zeros routinely. When zeros appear, the verdict needs higher-order information or a walk along the flat direction, not a louder restatement of the Hessian.

The Quick Version

A critical point has all partials zero at once; that is necessary for peaks and valleys, sufficient for nothing.
Hessian eigenvalues vote: all positive is a minimum, all negative a maximum, mixed a saddle, zero inconclusive.
For x squared minus y squared the votes are 2 and minus 2, a saddle you can walk in both directions.
In high dimensions split votes dominate, so training stalls at saddles far more often than at bad minima.
A small gradient with a bad loss means check curvature, not the learning rate.