Adam vs SGD with Momentum
How modern optimizers navigate the complex loss landscape to find the best model weights.
Verdict: Adam is the undisputed king for fast convergence and is the default for almost everything. SGD with Momentum is sometimes preferred by elite researchers to squeeze out the absolute highest accuracy on vision tasks.
The Short Answer
Optimizers are the algorithms that update the model's weights based on the gradient (the slope of the error).
- SGD with Momentum: Takes standard Stochastic Gradient Descent and adds physics. It imagines the optimizer as a heavy ball rolling down a hill. It builds up speed in consistent directions and resists sudden changes, helping it power through small bumps (local minima).
- Adam (Adaptive Moment Estimation): The industry standard. It not only uses momentum, but it also calculates a custom, individual learning rate for every single parameter in the model.
Where They Differ
| Feature | SGD with Momentum | Adam |
|---|---|---|
| Learning Rate | One global rate used for all weights. | Unique adaptive rate for every single weight. |
| Speed to Converge | Slower. Requires careful tuning of the learning rate. | Extremely fast. Highly forgiving of bad hyperparameter choices. |
| Final Accuracy | Often slightly higher on heavily tuned vision models. | Generally excellent, but can sometimes overfit or get stuck in "sharp" minima. |
| Memory Footprint | Moderate (stores 1 velocity value per weight). | Heavy (stores 2 moment values per weight). |
The Problem with a Global Learning Rate
Imagine a loss landscape shaped like a long, narrow canyon (a ravine). The walls are incredibly steep, but the floor slopes down very gently toward the optimum.
If you use standard SGD, the steep walls produce massive gradients. The optimizer will take a huge step, bounce off the wall, hit the other wall, and bounce back, violently oscillating without making much forward progress.
You could lower the global Learning Rate to stop the bouncing, but then your forward progress down the gentle slope of the canyon floor would become agonizingly slow.
How Adam Solves This
Adam fixes this by tracking the historical gradients for every single weight separately.
If Adam notices that Weight A (the steep wall) is producing massive gradients, it artificially shrinks the learning rate just for Weight A. If it notices that Weight B (the canyon floor) is producing tiny, consistent gradients, it artificially boosts the learning rate just for Weight B.
The result is that Adam stops bouncing and shoots smoothly down the center of the ravine.
Choose A When
(When to use SGD with Momentum)
- You are trying to win an image classification competition: For highly specific Computer Vision tasks (like training a ResNet on ImageNet), researchers often use heavily-tuned SGD. It has been empirically shown to find slightly "flatter" (more robust) minima than Adam, generalizing slightly better.
- You are willing to spend weeks tuning: Getting SGD to beat Adam requires meticulously hand-crafting Learning Rate Schedules (e.g., warming up, stepping down at epoch 30, cosine decay).
Choose B When
(When to use Adam)
- You are building almost anything else: Transformers, LLMs, GANs, tabular data, NLP, RL. Adam (or its variant AdamW) is the undisputed default.
- You want it to "just work": Adam is incredibly robust. The default learning rate of
1e-3or3e-4in PyTorch will usually get you 95% of the way to a perfect model without any hyperparameter tuning.
What People Get Wrong
Using vanilla Adam with L2 Regularization
If you want to use Weight Decay (L2 Regularization) to prevent overfitting, standard Adam actually implements it incorrectly due to how its adaptive math interacts with the penalty. You should almost always use the AdamW (Adam with decoupled Weight decay) optimizer instead. It is identical to Adam but fixes this math bug, resulting in better generalization.