Non-Convex Optimization Landscapes

The loss surface of a neural network is a high-dimensional terrain of valleys, saddles, and plateaus, and its shape decides which optimizer moves actually work.

Hessian eigenvalue signs sort each flat spot into a valley, a pass, or a plateau

Why Does This Exist?

A one-dimensional loss curve has two kinds of flat spots: bottoms and tops. A million-dimensional loss has vastly more kinds, and gradient descent visits them constantly. The practical question is never whether the surface is bumpy. It is which bump you stand on and what it permits next.

This page classifies the terrain: minima, maxima, saddles of every index, plateaus, and connected valleys. Neighbours cover the overview (advanced non-convex optimization), the escape moves (saddle-point escape), and the geometry view (loss landscape geometry).

The one idea you need first

The Hessian is the matrix of second derivatives. Its eigenvalues say how the surface curves along each principal direction: positive means bowl up, negative means dome down. At a critical point, those signs are the whole classification. See Jacobian and Hessian for the mechanics.

Think of It Like This

A pass between two peaks

Stand on a mountain pass. Face the two peaks and the ground curves down on both sides. Turn ninety degrees toward the two valleys and it curves up. One spot, two opposite curvatures, and your compass heading decides which you feel.

A saddle in a loss surface works the same way, except there are a million headings instead of two. Most directions curve up and a few curve down. Gradient descent feels the up directions and settles in, missing the exits entirely.

Where the analogy stops: on a real range you can see the peaks. In a loss surface you only get local curvature numbers, never a view.

How It Actually Works

At a critical point with Hessian $H$ :

All eigenvalues positive: minimum. Every direction curves up.
All negative: maximum. Every direction curves down.
Mixed signs: saddle. The count of negative eigenvalues is its index.
Zeros present: degenerate or flat direction. Higher-order terms decide.

Two empirical facts matter more than the taxonomy. First, in large networks saddles vastly outnumber true minima at high loss, so early training navigates passes, not valleys. Second, good minima come in connected valleys rather than isolated pits: you can often walk from one solution to another along a low-loss path, which is why averaging checkpoints works.

Worked example

Compare three surfaces at the origin:

$f(x,y) = x^2 + y^2$ : Hessian $\text{diag}(2,2)$ , eigenvalues $2, 2$ . Minimum. Gradient step $[-2x, -2y]$ points inward from $(1,1)$ as $[-2,-2]$ .
$f(x,y) = x^2 - y^2$ : Hessian $\text{diag}(2,-2)$ , eigenvalues $2, -2$ . Saddle of index 1. From $(1, 1)$ the negative direction is pure $y$ : stepping $y \to 1.1$ drops loss by about $0.21$ while $x$ is fixed.
$f(x,y) = x^2$ : Hessian $\text{diag}(2,0)$ , eigenvalues $2, 0$ . Valley: $y$ is flat and any $y$ with $x = 0$ is equally good. Computed at $(1, 5)$ : loss $1$ , gradient $[2, 0]$ , and moving $y$ from $5$ to $-3$ changes nothing.

Watch Out For

Counting saddles in low dimensions and extrapolating

In two dimensions saddles look rare and minima look common. In a million dimensions the combinatorics flip: there are far more ways to mix signs than to make every sign positive.

Teams that debug in 2D visualizations conclude saddles are a non-issue. Train a real network and the stall points are saddles far more often than minima.

Reading sharpness from a single Hessian eigenvalue

The largest eigenvalue (sharpness) gets quoted as the whole story. A valley can be sharp in one direction and flat in 999,999 others, and generalization tracks the bulk, not the max.

Look at the eigenvalue density or the trace, not just the top value, before calling a minimum sharp.

The Quick Version

Hessian eigenvalue signs classify every critical point into minimum, maximum, saddle, or flat valley.
High-dimensional surfaces hold far more saddles than minima at high loss values.
Good solutions sit in connected low-loss valleys, which is why checkpoint averaging helps.
Flat directions with zero eigenvalues mean whole subspaces are equally good.
Judge sharpness by the eigenvalue bulk, not by the single largest value.