Stochastic Gradient Descent, Visually

Full-batch gradient descent computes the exact empirical risk gradient across the entire dataset. When you have one million training examples, computing a single parameter update requires accumulating forward and backward passes across all one million rows. Even on modern accelerator hardware, a single step can take dozens of seconds. After that full pass, the parameter vector moves by a tiny fraction and the loss barely budges (from 2.400 down to 2.394). Exactness costs a full dataset pass per nudge, crawling in clock time while compute burns.

Guess Now, Correct Later

Pure Stochastic Gradient Descent (SGD) breaks the dataset bottleneck by picking a single random row and updating parameters immediately. Drawing sample row 7 takes merely 0.04 milliseconds to evaluate—nearly a million times faster than reading the full dataset. Although row 7 yields a noisy local slope of 3.10 instead of the true population gradient of 4.00, applying learning rate 0.25 moves the weights by 0.78 at once. Because the mathematical expectation of random sample gradients equals the true population gradient, SGD is an unbiased estimator: wrong on every single instant step, but exactly right on average over the epoch.

Steps Halve Nicely

On a standard quadratic loss function $L(w) = w^2$, the derivative is $\nabla L = 2w$. With learning rate $\eta = 0.25$, the update rule contracts neatly: $w_{t+1} = w_t - 0.25(2w_t) = 0.50 w_t$. Starting at weight 2.0 (loss 4.0), the initial step takes gradient 4.0 and jumps by 1.0 to land precisely on 1.0. From weight 1.0, the gradient drops to 2.0, jumping by 0.5 to land on 0.5. The next jump shrinks to 0.25. As the landscape flattens near the minimum, step lengths halve geometrically, ensuring smooth contraction without overshoot.

Average a Handful

Single-row SGD updates immediately, but its high variance causes steps to spray erratically in every direction. Full-batch descent is perfectly stable, but produces only one update per epoch. Mini-batch SGD balances both worlds: averaging 32 examples shrinks gradient variance by a factor of 32 ($\sigma^2 / 32$, reducing standard deviation by over 5.6×). The orthogonal noise vectors cancel each other out while the true downhill signal reinforces. In a dataset of one million rows, a batch size of 32 delivers 31,250 steady parameter updates per pass while saturating GPU tensor cores.

Noise Jumps the Rut

Deep learning loss landscapes are non-convex, riddled with shallow ruts and suboptimal local minima. Deterministic batch gradient descent rolls down the steepest path and halts permanently inside the first shallow basin at loss 0.80 because the true gradient hits zero. In contrast, mini-batch gradient noise acts as an implicit stochastic perturbation. These random kicks provide the kinetic energy needed to hop over shallow potential barriers, allowing SGD to bypass local traps and descend into the deep global basin near loss 0.10.

Calm the Noise

While gradient noise helps escape early traps, persistent noise prevents the optimizer from settling once it reaches the bottom of the basin. To guarantee convergence, practitioners use two complementary levers:

1. Learning rate annealing ($\eta \downarrow$): Halving the learning rate on validation plateaus shrinks the random walk radius directly ($r \propto \eta \sigma$), bringing parameters to a smooth rest.
2. Batch size expansion ($B \uparrow$): Growing the batch size from 32 to 128 and 512 suppresses gradient variance ($\sigma^2 / B$) while maximizing parallel GPU throughput.

The standard tactical sequence is clear: decay the learning rate first to contract jitter, and widen batch size second to maximize hardware efficiency.

Where It Breaks

If you maintain a large, fixed learning rate without decay, SGD encounters its fundamental failure mode. Near the true minimum, the expected gradient approaches zero, but individual sample variance does not vanish ($\text{Var}(g_i) = \sigma^2 > 0$). Every update delivers a full stochastic kick, forcing the optimizer into a persistent limit cycle orbiting the bottom. Instead of reaching zero loss, the weights bounce perpetually within a noise ball around loss 0.10. Without scheduled decay satisfying the Robbins-Monro conditions ($\sum \eta_t = \infty$ and $\sum \eta_t^2 < \infty$), noisy steps cannot land.

The Quick Version

• Full-batch passes crawl, spending millions of evaluations for one tiny parameter nudge.
• Single-sample SGD updates in 0.04 milliseconds—wrong on each row, but unbiased on average.
• On quadratic bowls, a 0.25 rate halves the distance to the minimum on every step.
• Mini-batches of 32 reduce gradient variance by 32× while delivering 31,250 updates per pass.
• Gradient noise acts as thermal excitation, hopping shallow local minima at 0.8 toward 0.1.
• Decay learning rate on plateaus to shrink jitter, and expand batch sizes to sustain throughput.
• Constant learning rates without decay orbit the basin in a persistent noise ball and never settle.