Stochastic Gradient Descent

Update the model after each example instead of after the whole dataset. The noise in those single-example steps shakes training out of flat spots where exact gradients stall.

From one start, the exact full-batch path slides into the nearest shallow dip and stops, while the noisy one-example path jitters out of that dip and reaches the deeper basin

Why Does This Exist?

Exact gradients cost a full pass per step. You score every row, average the slopes, then move once. On a dataset with millions of rows that means one update costs millions of forward passes, so training crawls even before anything goes wrong. And patience does not fix the second problem. At a saddle the averaged slope sits near zero, so the exact step is near zero and training waits in place.

Stochastic gradient descent pays per example instead. Score one drawn row, step at once, draw again. Two words to keep apart. A step is one parameter update. An epoch is one full pass over the data (see epoch). Batch learning takes one step per epoch. This takes one step per example, so it moves thousands of times while the exact method is still averaging.

This page covers the single-sample update and why its noise helps. The exact update it replaces lives in gradient descent, and the adaptive step sizes people pair with it live in Adam.

Think of It Like This

A ball rolling down a bumpy gutter while someone taps the side

Picture a gutter with dents of different depths and a ball rolling along it. Left alone, the ball drops into the first dent it meets and stays there, even when a deeper channel waits further along. Now tap the side of the gutter at odd moments as the ball rolls. Light taps knock it out of shallow dents but cannot lift it from a deep one, so it keeps travelling until it settles somewhere deep.

Each tap is one example disagreeing with the average. The taps are the mechanism, not sloppiness.

Where the picture stops: real taps are not random knocks. Each one follows the slope of the example just drawn, so the jitter shrinks as the fit improves and the ball calms down on its own near a good answer.

How It Actually Works

The update and what each symbol means

The rule looks like batch learning with one swap:

\theta_{t+1} = \theta_t - \eta \nabla_\theta L_i(\theta_t)

Here $\theta_t$ holds the parameters at step $t$ , $\eta$ is the learning rate that sets how far each step reaches, $L_i$ is the loss on a single drawn example $i$ , and $\nabla_\theta L_i$ is that example gradient with respect to $\theta$ . Draw $i$ uniformly at random and this one-example gradient matches the full-dataset gradient on average. Right in expectation, wrong in almost every single instance, and that wrongness does the work.

At a saddle or inside a shallow basin the averaged gradient is near zero but any one example gradient is not, so the step still moves and the next draw pulls a different way. Over time the jitter also favors wide basins. A narrow dip cannot hold a jittery run while a broad flat floor can, and flat floors tend to behave better on new data.

Worked example with four points

Fit one weight $w$ with no bias and prediction $w x$ on four points $(1, 2)$ , $(2, 3)$ , $(3, 5)$ , $(4, 7)$ . Use mean squared error, whose per-example gradient is $2x(wx - y)$ . Stand at $w = 1.0$ :

Point 1: $2 \times 1 \times (1 - 2) = -2.0$
Point 2: $2 \times 2 \times (2 - 3) = -4.0$
Point 3: $2 \times 3 \times (3 - 5) = -12.0$
Point 4: $2 \times 4 \times (4 - 7) = -24.0$

The full-batch mean is $(-2 - 4 - 12 - 24) / 4 = -10.5$ . Draw point 3 alone and the step sees $-12.0$ : close to the truth, off by $1.5$ , computed from a quarter of the data. With $\eta = 0.05$ the single-sample step lands at $1.0 + 0.05 \times 12 = 1.6$ , while the exact step lands at $1.0 + 0.05 \times 10.5 = 1.525$ . Same direction, slightly different distance, far less reading.

Batches, noise, and the learning rate

A small batch keeps most of the speed and trims the jitter. Noise falls with the square root of the batch size, so a batch of 4 carries about half the noise of single samples and a batch of 16 about a quarter. Very large batches drift back toward exact descent and lose the effect that was quietly helping, which is why the rate usually grows with the batch. Doubling the batch and doubling the rate holds the movement per epoch roughly steady. That scaling holds over a wide range and then stops past a model-specific point where extra rows buy almost nothing. You find that point by measuring, not by guessing.

Shuffling is part of the method, not tidiness. If the file is grouped by class, ten thousand same-label steps in a row pull one way before the other labels ever answer, and the run drifts instead of jittering. Shuffle every epoch. The tell for a missing shuffle is a loss curve that ripples once per epoch.

Code

The check below recomputes the worked example: per-sample gradients, their mean, and the two next weights.

1xs = [1.0, 2.0, 3.0, 4.0]2ys = [2.0, 3.0, 5.0, 7.0]3w = 1.04
5grads = [2 * x * (w * x - y) for x, y in zip(xs, ys)]6full = sum(grads) / len(grads)7print(grads)  # -> [-2.0, -4.0, -12.0, -24.0]8print(full)  # -> -10.59
10eta = 0.0511print(w - eta * grads[2])  # -> 1.612print(w - eta * full)  # -> 1.525

The single draw gives $-12.0$ against the true $-10.5$ , and the two updates land a small step apart, exactly as the arithmetic above says.

Watch Out For

Grouped data that never got shuffled

Symptom: the loss ripples once per epoch, and two runs with different seeds differ more than seed noise explains. Streaming feeds hide this well, since records arrive grouped by day or source and never pass a shuffle at all.

Fix: shuffle the dataset before every epoch. For a stream, hold a buffer of a few thousand records and sample from the buffer instead of reading in arrival order. Compare once against a shuffled baseline and the drift shows itself.

A bigger batch with the old learning rate

Symptom: you raise the batch to fill the hardware, epochs run faster, training loss looks fine, and validation comes back slightly worse. Nothing errored, so nobody investigates.

Fix: scale the rate up with the batch, warm it up over the first few hundred steps so the larger rate does not blow up early, and judge by validation rather than training loss. If the gap survives at the scaled rate, you are past the useful batch size and the throughput win is not free.

The Quick Version

SGD computes the gradient from one example or a small batch, so it steps once per draw instead of once per pass.
Each noisy gradient matches the true one on average, and the individual error walks training out of saddles and shallow dips.
The jitter favors wide flat basins, which tend to behave better on new data.
Noise falls with the square root of batch size, so grow the learning rate with the batch until measurements say stop.
Shuffle every epoch, or grouped data turns the noise into a drift with a once-per-epoch ripple.