Decoupled Weight Decay Regularization (AdamW)

The Problem

Before AdamW, the deep learning community widely believed that $L_2$ regularization and weight decay were mathematically equivalent operations. Most popular frameworks, including PyTorch and TensorFlow, implemented them interchangeably. When researchers wanted to apply weight decay to Adam, they simply added the $L_2$ penalty to the loss function.

However, Adam's generalization performance frequently lagged behind classic Stochastic Gradient Descent (SGD) with momentum, particularly on image classification tasks. The problem wasn't Adam's learning rate adaptation; it was how that adaptation interacted with the $L_2$ penalty. Because Adam scales gradients based on their historical variance, adding the $L_2$ penalty to the loss meant the weight decay was also being scaled by the optimizer. Parameters with large historical gradients received less regularization, while parameters with small historical gradients were heavily penalized.

The Idea

Loshchilov and Hutter made a critical realization: while $L_2$ regularization and weight decay are equivalent for standard SGD, they are not equivalent for adaptive gradient algorithms.

Their proposal, AdamW, is startlingly simple: stop adding the penalty to the loss function. Instead, decouple the weight decay step from the gradient update step. First, calculate the standard Adam gradient update (without the $L_2$ penalty). Then, manually decay the weights by a constant factor.

How It Works

The AdamW mechanism separates the optimization of the loss from the regularization of the weights:

1. The standard gradient calculation The optimizer computes the gradients of the pure loss function, ignoring any regularization terms. It updates the first and second moment estimates (the exponential moving averages of the gradients and their squares) exactly as standard Adam does.

2. The adaptive update The weights are updated using the computed moments and the current learning rate. At this point, the update is identical to Adam running without any regularization.

3. The decoupled weight decay Finally, the optimizer applies weight decay directly to the weights, completely independently of the adaptive gradient scaling. The weight decay factor is multiplied by the learning rate (and optionally a schedule multiplier) but is never divided by the gradient variance.

Why It Mattered

AdamW solved Adam's generalization problem. By decoupling the weight decay, the optimizer could reliably regularize all parameters equally, regardless of their historical gradient variance.

This simple fix allowed AdamW to match or beat SGD with momentum on tasks where standard Adam had historically failed, such as training convolutional neural networks for image classification. It provided the best of both worlds: the fast convergence of adaptive learning rates and the strong generalization of proper weight decay.

What Came After

AdamW became the default optimizer for modern deep learning. It is the standard choice for training Large Language Models (LLMs), Vision Transformers (ViTs), and diffusion models. Almost every major foundational model trained today—including GPT, Llama, and Stable Diffusion—uses AdamW or a closely related variant. Its success also forced the community to be more precise about the mathematical differences between loss penalties and update rules.