Adam Optimizer, Visually
Five pictures: Adam combines momentum and variance tracking to automatically tune the learning rate for every parameter, shooting straight down narrow valleys.
When optimizing a model, some parameters sit on steep slopes while others lie on flat plains. A single, global learning rate cannot satisfy both. If the step size is large, the steep parameters bounce out of control; if it is small, the flat parameters crawl to a halt.
Two Moving Averages
Adam assigns every parameter its own learning rate using two memory vectors. The first moment tracks momentum (the direction of recent gradients), while the second moment tracks variance (the squared magnitude of recent gradients).
Bias Correction
Because memory vectors start at zero, early estimates are severely biased. Adam scales up these early steps via bias correction, ensuring the optimizer behaves accurately right from the first update rather than starting artificially small.
Adaptive Steps
Dividing momentum by the variance's square root normalizes updates. Steep, noisy directions are automatically penalized with smaller steps, while flat, consistent directions are granted larger ones. The result is a precise trajectory straight toward the minimum.
Where It Breaks
Adam's aggressive, localized precision is a double-edged sword. It can converge extremely fast, but it frequently dives straight into sharp, narrow minima. These sharp valleys often overfit the training data and generalize poorly to unseen examples, whereas simpler optimizers like plain momentum tend to bounce into wider, more robust basins.
The Quick Version
- A single learning rate fails when parameters have drastically different scales.
- Adam tracks both momentum (direction) and variance (scale) per parameter.
- Bias correction forces the initial zero-state memory to be accurate immediately.
- Dividing by the variance auto-tunes the step size, shooting straight down valleys.
- The honest failure mode: Adam can quickly trap itself in sharp, poorly-generalizing minima.