Gradient Descent Variants

Full-batch, mini-batch and single-example steps use the same update rule and differ only in how many examples each gradient averages over.

From w equals 0 with learning rate 0.1, full-batch mean minus 6 lands at 0.6, mini-batch mean minus 7.5 lands at 0.75, and single-example minus 8 lands at 0.8, with 1, 313 and 10000 updates per epoch

Why Does This Exist?

Full-batch gradient descent is exact and slow. On $N = 10{,}000$ images it reads all 10,000 gradients, averages them, and takes one step per epoch. Each step points the right way, but an epoch buys exactly one update, and the whole dataset must fit through memory before anything moves.

Single-example SGD is the opposite: one example, one gradient, one step. An epoch buys 10,000 updates, and training starts learning after the first row. But each gradient is a noisy guess at the true direction, so the path zigzags and the loss jumps up as often as down from step to step.

Mini-batch splits the difference and is what most training actually runs. With $B = 32$ on the same 10,000 examples, an epoch buys $\lceil 10{,}000 / 32 \rceil = 313$ updates, each averaged over 32 rows. Noise falls roughly with $1/B$ while hardware stays busy, which is why 32 to 512 is the standard range. This page is about that batch-size choice only. The gradient itself is the prerequisite object; momentum and per-coordinate adaptation live in stochastic gradient descent variants; the step size schedule lives in learning rate schedules.

The one idea you need first

One update is always $\theta \gets \theta - \eta g$ , where $\theta$ holds the parameters, $\eta$ is the learning rate, and $g$ is whatever gradient estimate this variant computed. The variants differ only in $g$ : the mean over all $N$ rows, over a random $B$ rows, or over one row.

Think of It Like This

Surveying a hill by census, patrol, or lone scout

A full-batch step is a census: measure every plot on the hill, average, then take one confident stride. Exact, and you only move once the last form is counted.

Single-example SGD is a lone scout who phones in one reading and orders a step immediately. Ten thousand readings mean ten thousand steps, but any single reading can point the wrong way, so the march weaves.

Mini-batch is a patrol of 32 scouts reporting together. Each report averages out most of one scout's error, and reports arrive 313 times per sweep instead of once. The patrol costs 32 radios instead of 10,000, which is why expeditions use patrols.

Where it stops: scouts phone in independent readings, but training rows share the model and the data order, so consecutive batches correlate. Shuffle every epoch or the patrol walks in circles.

How It Actually Works

Write the true loss as the mean over rows and each variant as a mean over fewer rows. With per-example gradients $g_i$ :

g_{\text{batch}} = \frac{1}{N}\sum_{i=1}^{N} g_i, \quad g_{\text{mini}} = \frac{1}{B}\sum_{i \in \mathcal{B}} g_i, \quad g_{\text{SGD}} = g_i

In plain words: batch averages everything, mini-batch averages the $B$ rows in the current batch $\mathcal{B}$ , SGD takes one row as stands. All three then apply the same $\theta - \eta g$ . Cost per update scales with rows read ( $N$ , $B$ , $1$ ); updates per epoch scale inversely ( $1$ , $N/B$ , $N$ ); noise scales roughly inversely with rows averaged.

Worked example

Take four rows whose per-example gradients at $w_0 = 0$ are $[-8, -7, -5, -4]$ , with learning rate $\eta = 0.1$ .

Hand-compute each estimate. Full-batch mean: $(-8 - 7 - 5 - 4)/4 = -24/4 = -6.0$ . Mini-batch over the first two rows: $(-8 - 7)/2 = -15/2 = -7.5$ . Single-example on the first row: $-8.0$ . Same rule, three different $g$ values, all checked by addition and division.

Step each one from $w_0 = 0$ :

Batch: $w_1 = 0 - 0.1 \times (-6.0) = 0.6$ .
Mini-batch: $w_1 = 0 - 0.1 \times (-7.5) = 0.75$ .
SGD: $w_1 = 0 - 0.1 \times (-8.0) = 0.8$ .

The diagram above lands all three from the same start so the spread is visible: exact 0.6, patrol 0.75, lone scout 0.8. On this tiny $N = 4$ set the updates per epoch are 1, 2 and 4. Scale to $N = 10{,}000$ with $B = 32$ : batch gives 1 update, mini-batch gives $\lceil 10{,}000/32 \rceil = 313$ updates, SGD gives 10,000 updates. Exactness is bought with waiting.

Code

The point is counting updates and reproducing the three hand-computed steps from the same gradient list, not the formula retyped.

1def step(w: float, g: float, lr: float = 0.1) -> float:2    return w - lr * g3
4
5def updates_per_epoch(n: int, batch: int) -> int:6    return (n + batch - 1) // batch  # ceil division, no floats7
8
9grads: list[float] = [-8.0, -7.0, -5.0, -4.0]10g_batch = sum(grads) / len(grads)11g_mini = sum(grads[:2]) / 212g_sgd = grads[0]13print([g_batch, g_mini, g_sgd])  # -> [-6.0, -7.5, -8.0]14print([step(0.0, g) for g in (g_batch, g_mini, g_sgd)])  # -> [0.6, 0.75, 0.8]15print(updates_per_epoch(10000, 10000))  # -> 116print(updates_per_epoch(10000, 32))  # -> 31317print(updates_per_epoch(10000, 1))  # -> 10000

The first two lines match the hand arithmetic exactly, and the last three show why batch size is a throughput choice: 1, 313 or 10,000 chances to move per pass over the same data.

Watch Out For

Comparing one SGD step against one batch step

A single SGD step looks worse than a batch step almost every time, because one noisy row loses to the exact mean. Teams then conclude SGD is broken and switch back to full-batch, recovering smooth steps and losing the 313-to-1 update advantage.

Compare per epoch, not per step. Fix the epoch budget, watch the loss after each full pass, and let the frequent noisy steps accumulate before judging. The noise averages out across updates the way the batch averages it within one.

Reusing the full-batch learning rate for small batches

A learning rate tuned for exact batch gradients is usually too large for noisy single-row gradients. The symptom is a loss that falls on batch and explodes the moment training switches to SGD or a small mini-batch, which looks like broken data but is just oversized steps on noisy estimates.

Retune when the batch size changes: smaller batches usually want a smaller rate, and shuffling every epoch keeps consecutive batches from repeating the same bias. Change one of batch size, rate or shuffle at a time so the cause stays identifiable.

The Quick Version

One rule for all three: $\theta \gets \theta - \eta g$ , where only the gradient estimate $g$ changes.
Full-batch averages all $N$ rows: exact direction, 1 update per epoch, waits on all data.
SGD uses one row: 10,000 updates per epoch on 10,000 rows, each step noisy.
Mini-batch with $B = 32$ takes 313 steps per epoch on the same data, the standard balance.
From $w_0 = 0$ with $\eta = 0.1$ , means $-6.0$ , $-7.5$ and $-8.0$ land at $0.6$ , $0.75$ and $0.8$ .
Compare variants per epoch and retune the rate when the batch size changes.