Hessian Matrix

Where the gradient holds first derivatives, the Hessian holds second ones: entry $i, j$ is $\partial^2 f / \partial w_i \partial w_j$. It is square, symmetric for well-behaved functions, and it describes curvature — whether the surface bends upward, downward, or both at once.

Curvature is what tells a flat spot apart. At a point where the gradient is zero, an all-positive-curvature Hessian means a minimum, all-negative means a maximum, and a mix of both means a saddle. In high-dimensional loss landscapes saddles vastly outnumber local minima, which is a genuinely useful thing to know about why training gets slow rather than stuck.

Second-order methods use the Hessian to take much better-informed steps than gradient descent, and they are still rare in deep learning for one reason: a model with $n$ parameters has an $n \times n$ Hessian. At 7 billion parameters that is $4.9 \times 10^{19}$ entries. Adam and friends are cheap diagonal approximations to this information.