Adaptive Optimizers

Why Does This Exist?

Picture fine-tuning a text classifier. The embedding layer's gradients trickle through while the final layer's gradients arrive loud, easily tens of times apart in size. You get one global learning rate to serve both. Set it for the loud layer and the quiet one barely moves for epochs. Set it for the quiet one and the loud one jumps past anything useful on the first steps.

That mismatch is the whole reason this family exists. Scale makes it hurt more: two state vectors per parameter at 4 bytes each turns 7 billion parameters into about 56 billion bytes of optimizer state before weights or gradients take a byte. So the choice of how you track per-parameter history stops being trivia and starts deciding whether the run fits at all.

This page maps that family: where AdaGrad's accumulation stalls, how RMSprop fixes it by forgetting, how Adam pairs the fix with momentum, and which answer fits which cost, Adafactor for memory, Lion for compute, LAMB for very large batches. The full Adam update with bias correction and AdamW lives in Adam and AdamW. The loop all of these plug into lives in Gradient Descent.

The one idea you need first is that loop's step. Parameters $\theta$ move against the gradient $g$ by a global rate $\eta$: $\theta$ minus $\eta$ times $g$, repeated. Momentum improves it by averaging recent gradients so consistent direction stacks up. Adaptive methods attack the other half of the problem, the size of the step per parameter, and they stack with a global schedule such as Learning Rate Scheduling.

Think of It Like This

How It Actually Works

Every member shares one move: divide each parameter's gradient by a measure of how large that parameter's gradients have been running. Quiet parameters get boosted, loud ones get held back, and one global $\eta$ finally suits the whole model. The second moment is that measure, an average of squared gradients kept separately for every parameter.

The shared rule

Keep one running average of the squared gradient per parameter:

Here $g_t$ is the gradient at step $t$, $v_t$ is the second moment after seeing it, and $\beta$ is a decay between 0 and 1, often 0.9, that sets how fast old squares are forgotten. Then step with the gradient divided by its own recent size:

Here $\theta_t$ holds the parameters at step $t$, $\eta$ is the single global learning rate, and $\epsilon$ is a tiny guard, typically $10^{-8}$, that keeps you from dividing by something near zero when a parameter has sat still. A quiet parameter owns a small $\sqrt{v_t}$, so the division boosts it. A loud one owns a large $\sqrt{v_t}$, so the same division shrinks it.

From AdaGrad to RMSprop to Adam

AdaGrad came first and never forgets. It sums every squared gradient, $G_t = G_{t-1} + g_t^2$, and steps $\eta$ times $g_t$ over $\sqrt{G_t}$ plus $\epsilon$. Rare features start with a near-zero sum, so their early steps are large, which made AdaGrad loved for sparse text data. But the sum only grows, so on a long dense run every step shrinks toward zero and training stalls mid-run. Good opener, poor finisher.

RMSprop fixes the stall by swapping the sum for the moving average above, with $\beta$ around 0.9. Old squares fade, the denominator tracks recent size instead of all-time size, and steps stop decaying. One state vector per parameter, no momentum, still a sane pick when memory is tight and you doubt momentum buys much.

Adam adds the first moment, the momentum average $m_t$ with its own decay $\beta_1$ usually 0.9, alongside the second moment with $\beta_2$ usually 0.999. Direction comes from $m_t$, scale from $v_t$, and both get bias correction for their zero start before the division. Two states per parameter. That correction and the AdamW decoupling are worked fully in Adam and AdamW, so this page treats Adam as the default the rest of the family is measured against.

Adafactor, Lion, and LAMB answer different costs

Adafactor answers memory. For an $n$ by $m$ weight matrix, Adam's two states are two full $n$ by $m$ arrays. Adafactor keeps one row factor of length $n$ plus one column factor of length $m$ and rebuilds each entry's scale from their outer product. Take a 4096 by 4096 layer in float32: Adam holds 2 times 4096 times 4096 times 4 bytes, about 134.22 MB, while the factors hold 4096 plus 4096 times 4 bytes, about 0.0328 MB, roughly 4096 times smaller. Coarser history, far cheaper. Reach for it when optimizer state is what stops the run fitting.

Lion answers per-step compute. It keeps one momentum-like average and steps by its sign rather than its size, so every parameter moves the same distance and only direction differs, with decay applied outside the sign. One state, cheaper arithmetic, but the magnitude information Adam keeps is gone. Treat a switch as a fresh tuning job, never a drop-in.

LAMB answers very large batches. It takes an Adam-style raw update $u$ for each layer with weights $w$ and rescales it by the layer's trust ratio, the norm of $w$ over the norm of $u$, so every layer moves in proportion to its own size no matter how the giant batch pushed the raw gradients. Same direction Adam chose, different distance per layer.

The trade is plain: adaptivity buys you one rate that works across wildly scaled layers, and charges memory, extra arithmetic, and sometimes a little generalization on vision tasks where tuned SGD with momentum still wins. Skip the family on small clean convex fits, where plain descent plus a schedule is simpler and every bit as good. The shared assumption is that recent squared size predicts the right scale right now. It fails when gradients spike suddenly against a stale denominator or wake from near zero, which is when $\epsilon$ and warmup earn their keep, see Learning Rate Scheduling.

Worked example: two gradients, one step size

Fix $\eta = 0.01$, $\beta = 0.9$, $\epsilon = 10^{-8}$, and take one RMSprop step from $v_0 = 0$ for a quiet parameter with $g = 0.1$ and a loud one with $g = 2.0$, twenty times larger. Square them: 0.01 and 4.0. Average against zero: $v$ becomes 0.1 times 0.01, which is 0.001, and 0.1 times 4.0, which is 0.4. Square-root them: about 0.0316 and about 0.6325.

A 20 times gap in gradients becomes the same 0.032 step after the division, which is the picture above. Note the first-step overshoot: bias correction would divide each $v$ by 0.1 and land both steps at exactly 0.01, so the uncorrected 0.032 runs about 3.16 times large. That early sizing error is why Adam and AdamW corrects before dividing.

Code

This snippet replays the worked example so you can check the equal steps yourself:

1import math2
3
4def rms_step(g: float, v_prev: float = 0.0, beta: float = 0.9,5             lr: float = 0.01, eps: float = 1e-8) -> tuple[float, float]:6    v = beta * v_prev + (1.0 - beta) * g * g  # recent squared size7    step = lr * g / (math.sqrt(v) + eps)       # scaled step for this param8    return v, step9
10
11for g in (0.1, 2.0):12    v, step = rms_step(g)13    print(f"g {g:.1f}  v {v:.4f}  step {step:.4f}")14    # -> g 0.1  v 0.0010  step 0.031615    # -> g 2.0  v 0.4000  step 0.0316

Both print 0.0316. Change beta to 1.0 and watch the second call shrink on repeat steps: that is AdaGrad's stall in one line.

Watch Out For

The Quick Version

• Adaptive methods divide each gradient by its own recent size, so one global rate suits layers whose gradients differ tens of times over.
• AdaGrad sums all past squares and stalls on long runs; RMSprop swaps the sum for a forgetting average and keeps one state.
• Adam pairs that scale with a momentum direction plus bias correction, holding two states, which is why it is the default.
• Adafactor rebuilds the second moment from row plus column factors, about 4096 times smaller than full state on a 4096 by 4096 layer.
• Lion steps by the sign of one average, cheaper per step but blind to magnitude, so it needs its own tuned rate.
• LAMB rescales each layer's update by its weight norm over its update norm to hold together at very large batches.

What to Read Next

• Adam and AdamW
• Gradient Descent
• Momentum
• Learning Rate Scheduling