Local vs Global Minima

A local minimum beats its neighbours; the global minimum beats everything. Training usually settles for local, and that turns out fine.

The W shaped quartic traps a descent that starts right of centre in the right valley at minus 0.25 while the left valley stays unreachable without climbing over the hill

Why Does This Exist?

"Did training find the best weights?" has two readings. The global minimum is the best over the entire landscape; a local minimum is merely better than everything nearby. Gradient descent follows local slopes, so it can only promise something local. The gap between those promises is where a lot of optimisation anxiety lives.

Here is the good news the theory eventually delivered. Finding global minima of general non-convex functions is genuinely hard (NP-hard in the worst case), yet deep networks train well anyway. Three facts explain why: in high dimensions most critical points are saddles rather than traps, many distinct valleys perform about equally well, and valleys connect through low-loss paths more often than the one-dimensional pictures suggest. The local-versus-global distinction matters enormously in low dimensions and surprisingly little at scale.

The search procedure that finds the candidates, flat slopes plus curvature tests, is taught in critical points, maxima and minima. The landscape story around them, saddles and plateaus, continues in convex and non-convex optimisation.

The one idea you need first

A point $x^\star$ is a local minimum if $f(x^\star) \le f(x)$ for all $x$ in some neighbourhood around it, and a global minimum if the inequality holds for every $x$ in the domain. Every global minimum is local; the reverse fails exactly when a second, deeper valley exists elsewhere.

Think of It Like This

Two lakes at different heights in fog

You stand in foggy mountains with an altimeter, tasked with finding the lowest lake. You walk downhill until the ground rises in every direction: a lakeshore. That is a local minimum.

But fog hides the next valley, whose lake sits fifty metres lower. Your altimeter proves your lake is the lowest nearby, and says nothing about the next valley. To check globally you would have to climb out, cross the ridge and descend again, paying altitude to gain information.

High-dimensional training is fog with a twist: there are so many directions that a point which looks like a lakeshore almost always has some direction still descending. True traps are rare; passes (saddles) are everywhere. The fog lifts differently than your three-dimensional instincts expect.

How It Actually Works

Take $f(x) = x^4 - 3x^2 + 2$ and find every candidate:

f'(x) = 4x^3 - 6x = 2x(2x^2 - 3)

Flat slopes at $x = 0$ and $x = \pm\sqrt{1.5} \approx \pm 1.225$ . The second derivative $f''(x) = 12x^2 - 6$ sorts them: $f''(0) = -6 < 0$ , so the middle point is a local maximum with $f(0) = 2$ . At the outer points $f''(\pm 1.225) = 12 > 0$ , both valleys, with values:

f(\pm\sqrt{1.5}) = 2.25 - 4.5 + 2 = -0.25

Both valleys tie at $-0.25$ , so both are global minima. A descent starting at $x = 2$ rolls into the right valley and stays there; reaching the left one requires climbing over the hill at 0. Same value, unreachable from each other by pure descent. That is the whole phenomenon in one line: local methods find a valley, not the valley.

Code

The point is the check: descents from different starts land in different valleys of equal value.

1def descend(start: float, lr: float = 0.1, steps: int = 200) -> float:2    x = start3    for _ in range(steps):4        x = x - lr * (4 * x**3 - 6 * x)  # gradient of x4 minus 3x2 plus 25    return x6
7
8print(round(descend(2.0), 4))   # -> 1.22479print(round(descend(-2.0), 4))  # -> -1.2247

Both runs converge to $\pm\sqrt{1.5} \approx \pm 1.2247$ with value $-0.25$ . Two global minima, each invisible from the other side. Restarting found the twin; it did not find anything better, because nothing better exists.

Watch Out For

Reporting a converged run as the global best

Convergence proves the slope vanished, nothing more. In a landscape with valleys of different depths, one run reports one valley. If the depth matters (small models, scientific fitting, combinatorial objectives), restart from several widely spread starts and compare values instead of trusting a single trajectory.

Restarting with the same seed and calling it a restart

Two runs with identical initialisation, data order and augmentation follow identical trajectories to the identical valley. A restart ensemble needs genuine diversity: different seeds, different shuffles, ideally different architectures or data splits. Correlated restarts waste compute while looking diligent.

The Quick Version

Local minima beat their neighbours; the global minimum beats every point in the domain.
The quartic $x^4 - 3x^2 + 2$ has twin global valleys at $\pm 1.225$ with value $-0.25$ , split by a hill.
Pure descent finds a valley, not the valley; reaching another means climbing first.
Global optimisation is hard in general, yet deep networks train well because saddles dominate traps.
Restart with genuinely different seeds when depth matters; identical seeds re-find the same valley.