Momentum, Visually

When optimizing a model, the loss surface often takes the shape of a long, narrow ravine. Because plain gradient descent strictly follows the steepest immediate slope, it perpetually overcorrects. This forces the path to aggressively zig-zag back and forth across the valley walls rather than moving efficiently and directly down the center.

Remember your past

Instead of relying purely on the immediate slope, momentum remembers where you have been. It maintains a running total of recent gradients in a dedicated velocity vector. Every new gradient updates this velocity, ensuring the model retains a persistent memory of its recent trajectory and doesn't get completely derailed by a single sharp turn.

Vector addition

Because the steep valley walls slope in opposite directions, the sideways gradient pushes cleanly cancel each other out over time. Meanwhile, the smaller forward gradients—which point down the valley floor—share the same sign and continually accumulate. The mathematical result is that your velocity becomes a smooth, accelerated dive straight down the center.

Where It Breaks

That same beneficial, built-up velocity becomes a serious liability when you finally reach the bottom of the basin. A model carrying high momentum can simply fail to hit the brakes at the true minimum, overshooting the target completely and oscillating wildly before settling. You must always tune the momentum coefficient carefully alongside your learning rate.

The Quick Version

• Plain gradient descent wastes valuable computational steps by zig-zagging across narrow valleys.
• Momentum solves this by maintaining a running average of past gradients called velocity.
• Opposing sideways moves naturally cancel out, while consistent forward motion heavily stacks up.
• The honest failure mode: too much momentum causes the model to violently overshoot the true minimum.

What to Read Next

• Multi-Layer Perceptron
• Overfitting vs Underfitting