Adam Optimizer, Visually

The dashed projection separates the long horizontal movement from the short vertical movement in this cropped parameter-space view. The faint slanted lines are equal-loss contours for the first minibatch. Plain stochastic gradient descent (SGD) multiplies every gradient coordinate by the same learning rate, so the tenfold gradient difference remains a tenfold displacement difference.

Our synthetic example starts at weights (1, 1), with gradient (2, 0.2). At learning rate 0.1, SGD moves them by (−0.2, −0.02), ending at (0.8, 0.98). Adam uses each coordinate's history to rescale its update. Throughout this page, displacement means new weight minus old weight.

Keep Two Histories

The two branches keep different information. This drawing zooms into the x coordinate: the stored direction arrow is half the input arrow's length. On the right, the dashed square represents the raw squared gradient; the mint square has half its area and represents the stored squared-gradient average. Area, rather than side length, encodes the squared quantity.

Adam stores a signed-gradient average, m, and a squared-gradient average, v, per parameter. At update t, gₜ is the current gradient; the decay factors β₁ and β₂ set how much old history is kept. Squaring retains magnitude even when a gradient is negative:

$$m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t$$ $$v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2$$

The square applies separately to each coordinate. Start both averages at zero and choose decay factors β₁ = β₂ = 0.5 for readable arithmetic. After the first gradient, m = (1, 0.1) and v = (2, 0.02). These are teaching settings, not recommendations. The second moment isn't centered variance. The two histories cost two state values per weight.

Correct the Zero Start

Starting a moving average at zero pulls its early values toward zero. The raw/corrected comparison continues the x-coordinate detail: the direction arrow doubles in length, while the square doubles in area. Adam divides each average by its own correction factor:

$$\hat m_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat v_t = \frac{v_t}{1-\beta_2^t}.$$

At step one, both divisors are 0.5: corrected direction becomes (2, 0.2), and corrected second moment becomes (4, 0.04). This compensates for initialization, not sampling noise, and doesn't replace the running history with the current gradient at every later step.

Divide Each Coordinate

Read each row from its direction arrow, through the middle scale ruler, to the resulting step. The previous square encodes area, so its side length represents the square root used here. A larger remembered magnitude provides a larger divisor. Here, the large and small direction components become almost equal displacements after each is divided by its own scale. Compare x with y within each column; the output uses parameter units. Following Adam's update rule:

$$\Delta w_i = -\alpha\frac{\hat m_i}{\sqrt{\hat v_i}+\varepsilon}.$$

The global learning rate is α = 0.1. Choose ε = 0.01 outside the square root so its effect is visible. The x calculation is −0.1 × 2 / (2 + 0.01); the y calculation is −0.1 × 0.2 / (0.2 + 0.01). The first displacements are approximately (−0.09950, −0.09524), not exactly −0.1. The resulting weights, w₁, are approximately (0.90050, 0.90476). This isn't a universal rule that flatter coordinates travel farther.

When the Gradient Reverses

In this magnified x-coordinate view, the old and new arrows already include their half weights. Dashed guides align the endpoints; the memory arrow is their sum. The new gradient points the other way, but the retained contribution from the previous step is stronger.

Our next gradient is (−0.5, −0.05). Blending half the old signed average with half this gradient gives m = (0.25, 0.025), still positive in both coordinates.

The squared-gradient average becomes v = (1.125, 0.01125). At step two, both correction factors are 0.75. After correction and scaling, both weights decrease again, by approximately (−0.02700, −0.02516), reaching w₂ ≈ (0.87350, 0.87960). History can outweigh the latest gradient's sign; smoothing doesn't guarantee an instant change of direction.

The Rate Still Matters

The ring and diamond mark alternative endpoints from the same starting state, using learning rates 0.1 and 0.2. They use identical corrected histories. Doubling the rate doubles the displacement vector: approximately (−0.09950, −0.09524) becomes (−0.19900, −0.19048).

This comparison holds the histories fixed. In general training objectives, future gradients can change with the resulting weights, so whole trajectories needn't double. Choose settings using validation evidence; adaptation doesn't choose the global rate for you.

Where It Breaks

The picture plots the current minibatch's loss along the straight parameter-space segment from w₁ to w₂. The horizontal axis follows progress along this one update; height shows loss. The uphill movement shows the cost of retaining the old direction.

Our illustrative minibatches have linear losses: first 2wₓ + 0.2wᵧ, then −0.5wₓ − 0.05wᵧ. On the second, decreasing both weights increases loss. The plotted slice is straight because this loss is linear.

Before the second update, that batch's loss is approximately −0.49549; afterward it is −0.48073, an increase of 0.01476. A less negative loss is larger. This is a concrete limit on per-step descent, not proof of eventual divergence or poor generalization. These linear losses are local teaching examples, not a bounded training objective.

The Quick Version

• SGD preserves gradient scale differences in its displacements.
• Adam keeps signed-gradient and squared-gradient histories per parameter.
• Bias correction compensates for moments initialized at zero.
• Epsilon affects coordinatewise scaling, including the first step.
• Momentum can retain its sign after gradients reverse.
• Adam still needs a global learning rate.
• An Adam step can increase the current minibatch loss.