Momentum

Plain descent zig-zags across a narrow valley. Average the last few gradients instead, so sideways pushes cancel and the downhill push keeps stacking up.

Across-valley gradients of plus and minus 4.0 cancel near 3.1 while steady 0.4 along the floor builds velocity to 2.28 after 8 steps with beta 0.9

Why Does This Exist?

A heating bill model learns from floor area and room count, two inputs that move together. The loss surface becomes a long narrow trough, steep across and nearly flat along. Plain gradient descent points across the trough at almost every step, so it throws the weights at one wall and then the other. The small along floor component that actually lowers loss gets buried. Lowering the rate tightens the zig-zag but slows the crawl even more.

This page covers the averaging fix that lets the consistent direction win. The loop it plugs into lives in gradient descent. The noisy per batch version lives in stochastic gradient descent. The per weight scale built on top lives in adam optimizer.

Think of It Like This

A bobsled in an ice chute

A bobsled drops down a banked ice chute. Each wall shoves it toward the middle, first left then right, and those shoves alternate every second. Over a full run they cancel, so the sled travels down the middle without steering.

Gravity down the chute never flips. Every metre adds forward speed in the same direction.

That split is the mechanism. Alternating pushes cancel out of the motion while the steady pull accumulates into speed.

Where it stops: a sled has real friction and a driver with brakes, while velocity in an optimizer has neither. The sled also cannot sail uphill past the finish, but accumulated velocity can sail straight past a narrow minimum.

How It Actually Works

Call $\theta_t$ the weights at step $t$ , meaning the full parameter vector. Call $g_t$ the gradient $\nabla L(\theta_t)$ , meaning the uphill slope at those weights. Call $v_t$ the velocity, meaning a running sum of past gradients with the same shape as $\theta$ . Call $\beta$ the momentum coefficient between 0 and 1, and $\eta$ the learning rate:

v_{t+1} = \beta v_t - \eta g_t, \qquad \theta_{t+1} = \theta_t + v_{t+1}

At $\beta = 0$ this is plain descent. At $\beta = 0.9$ each old gradient keeps 90 percent of its pull each step, so the effective window is about $1/(1-\beta)$ , about 10 steps. A steady gradient settles at $\eta/(1-\beta)$ , about 10 times the plain step.

Worked example: cancel across, stack along

Take $\eta = 1$ and $\beta = 0.9$ . Across the valley the gradient alternates $4.0, -4.0$ for 8 steps. The velocity runs $-4.00, 0.40, -3.64, 0.72, -3.35, 0.99, -3.11, 1.20$ . It never exceeds the driving 4.0 because each new push fights nine tenths of the old one.

Along the floor the gradient is a steady $0.40$ , ten times smaller. The velocity runs $-0.40, -0.76, -1.08, -1.38, -1.64, -1.87, -2.09, -2.28$ . After 8 steps the small steady direction has built 2.28, closing on the 4.0 limit, while the large alternating direction sits at 1.20. That inversion is the diagram: one jagged path crawling, one smooth path running the floor.

Nesterov looks where velocity points

Standard momentum measures $g_t$ where the weights sit. Nesterov measures it at $\theta_t + \beta v_t$ , meaning the spot velocity is about to reach. When the floor starts rising there, the correction lands in this step rather than the next, so overshoot damps one step earlier.

Code

The two directions side by side with the exact update the optimizer runs.

1import numpy as np2
3def velocity(grads: np.ndarray, beta: float = 0.9) -> np.ndarray:4    v: float = 0.05    out: np.ndarray = np.empty(len(grads))6    for t, g in enumerate(grads):7        v = beta * v - float(g)8        out[t] = v9    return out10
11across: np.ndarray = np.tile([4.0, -4.0], 4)12along: np.ndarray = np.full(8, 0.4)13
14print(np.round(velocity(across), 2))  # -> [-4.    0.4  -3.64  0.72 -3.35  0.99 -3.11  1.2 ]15print(np.round(velocity(along), 2))  # -> [-0.4  -0.76 -1.08 -1.38 -1.64 -1.87 -2.09 -2.28]

The across row stays bounded near 3.1 while the along row grows to 2.28. Flip beta to 0.0 and both rows equal the raw gradients.

Watch Out For

Raising beta without cutting the rate

Symptom: a healthy run at beta 0.9 diverges within a few hundred steps after moving beta to 0.99. Nothing else in the config looks wrong.

Fix: effective step is about $\eta/(1-\beta)$ , so 0.9 to 0.99 multiplies reach by ten. Divide $\eta$ by ten at the same time, and treat momentum as part of step size rather than a separate dial.

Old velocity ruins a fine tune restart

Symptom: the rate drops to a tiny value for careful tuning, yet loss gets worse for the first few hundred steps before recovering. The tiny rate was supposed to make that impossible.

Fix: velocity still holds about 10 steps of large old gradients and pushes past the narrow optimum. Reset velocity to zero when the rate drops hard, or lower beta together with the rate.

The Quick Version

Plain descent in a narrow valley wastes most of every step going sideways across the walls.
Velocity averages recent gradients with decay beta, and the step follows velocity.
Beta 0.9 means an effective window near 10 steps and reach near 10 times the rate.
Alternating 4.0 across stays bounded near 3.1 while steady 0.4 along builds to 2.28 after 8 steps.
Nesterov scores the gradient where velocity points, so overshoot corrects one step earlier.
Cost is one extra parameter copy plus a new way to sail past a narrow minimum.