Skip to content
AI360Xpert
Paper Breakdowns
Paper breakdown

Adam Optimizer

Introduced Adam, an adaptive optimization algorithm that combined the best properties of momentum and RMSProp, becoming the default optimizer for deep learning.

Paper: Adam: A Method for Stochastic Optimization

Authors: Diederik P. Kingma, Jimmy Ba · 2014

Read the paper
Adam maintains per-parameter learning rates by tracking both the exponentially decaying average of past gradients (momentum) and squared gradients.
Adam maintains per-parameter learning rates by tracking both the exponentially decaying average of past gradients (momentum) and squared gradients.

The Problem

Training a neural network requires an optimization algorithm to update the weights based on gradients. Standard Stochastic Gradient Descent (SGD) applies the same global learning rate to all parameters. This is inefficient: features that appear frequently need small updates, while rare features need larger updates. Prior adaptive methods like AdaGrad and RMSProp tried to fix this, but they struggled with complex, non-convex loss landscapes or decayed their learning rates too aggressively.

The Idea

The authors combined the benefits of two existing extensions to SGD:

  1. Momentum, which keeps the optimizer moving in the same direction it was previously heading (like a ball rolling down a hill).
  2. RMSProp, which scales the learning rate for each parameter individually based on the recent magnitude of its gradients.

How It Works

Adam stands for Adaptive Momentum Estimation.

For every single parameter in the network, Adam calculates two values:

  1. The first moment (the mean): An exponentially decaying average of past gradients. This acts like Momentum.
  2. The second moment (the uncentered variance): An exponentially decaying average of past squared gradients. This acts like RMSProp.

Adam updates the parameter by taking the first moment (direction) and dividing it by the square root of the second moment (magnitude). It also includes a crucial 'bias correction' step to prevent these averages from being biased towards zero at the very beginning of training.

Why It Mattered

Adam is arguably the most widely used optimization algorithm in deep learning. It requires very little hyperparameter tuning (the default learning rate of 0.001 works for a massive range of problems), converges quickly, and handles sparse gradients beautifully. It democratized deep learning by making model training vastly more reliable and less dependent on expert intuition.

What Came After

Adam is still the undisputed king of optimizers. While some researchers occasionally use SGD with Momentum for extreme fine-tuning to squeeze out the last drop of generalization on specific image datasets, Adam (and its variant AdamW, which fixes how weight decay is applied) is the default choice for almost everything, including training massive LLMs.