Second-Order Optimization
Optimization that uses the Hessian, or an approximation of it, to account for curvature and take a more direct step than the gradient alone allows.
Think of It Like This
Like driving downhill with headlights that illuminate both the immediate slope and the upcoming curve, so you don't overshoot the hairpin turn.
Newton's method updates using the inverse Hessian times the gradient, converging on a purely quadratic function in a single step because the Hessian fully describes that surface's curvature. The Hessian of an n-parameter model is an n×n matrix, though, which is computationally out of reach at any modern model's scale — for 7 billion parameters that's on the order of 10^19 entries.
Quasi-Newton methods like L-BFGS approximate curvature from recent gradient history instead of computing the Hessian directly, and Adam's second moment is a cheap, diagonal-only stand-in for the same idea — a small fraction of what a true second-order method sees, but affordable at scale.