Batch Size Effects

How many examples you average into one update sets how noisy that update is. Small batches wobble more but take more steps per pass, while big batches are steadier yet each doubling buys less.

Sixteen times the examples per batch makes the gradient only four times steadier, while steps per epoch fall sixteen-fold.

Why Does This Exist?

A full-batch gradient over the whole dataset costs a full pass per step, so nobody trains that way on real data. Stochastic gradient descent instead estimates the gradient from a small random minibatch, and the batch size $B$ is the knob that sets how rough that estimate is.

The same knob pulls two other levers at once. On 1,024 training images, $B = 32$ gives 32 updates per pass while $B = 512$ gives only 2, so bigger batches mean far fewer chances to adjust. And past some size, very large batches often finish with worse test accuracy than smaller ones at the same training loss, which surprises everyone who expected steadier steps to always win.

This page covers that three-way trade: noise, step count, and test accuracy. The update rule itself lives on the stochastic gradient descent page, faking a big batch on a small GPU belongs to gradient accumulation, and the learning-rate half of scaling up belongs to learning rate warmup.

Quick grounding if SGD is new to you: each step moves the weights against the average gradient of the current minibatch, so the step points roughly downhill on the full loss but wobbles around the true direction. That wobble is the noise this whole page is about, and $B$ sets its size.

Think of It Like This

Taste-testing a crate of apples

You buy apples by the crate and want to know how sweet the crate is. Bite two apples and one mealy outlier ruins your verdict. Bite twenty and the odd bad apple barely moves the average, so your verdict steadies.

But each bite costs time. Twenty-apple verdicts mean you judge few crates per hour, and going from twenty apples to two hundred barely changes the verdict while costing ten times the chewing. So you pick a sample big enough to trust and small enough to keep moving.

Where it stops: apples never punish you for tasting too many. Neural nets sometimes do, because the wobble of small batches nudges training toward solutions that hold up on new data. That twist has no taste-test equivalent, and it is the subject of the mechanism section.

How It Actually Works

The minibatch gradient and its noise

Write $g_i$ for the gradient of the loss on example $i$ , and $B$ for the batch size. The minibatch gradient is the plain average:

g_B = \frac{1}{B} \sum_{i=1}^{B} g_i

Because examples are sampled at random, $g_B$ is unbiased: averaged over many batches it equals the full-dataset gradient. Its scatter is what matters. If per-example gradients scatter with standard deviation $\sigma$ , the average of $B$ of them scatters with standard deviation $\sigma / \sqrt{B}$ . Here $\sigma$ is per-example variability and $B$ is the example count in the batch. Doubling $B$ divides noise by $\sqrt{2} \approx 1.41$ , not by 2.

Worked example: sixteen times the examples, four times steadier

Say per-example gradients scatter with $\sigma = 2.0$ around a true gradient of $1.0$ . At $B = 4$ :

\text{noise} = \frac{2.0}{\sqrt{4}} = \frac{2.0}{2} = 1.00

so a typical step aims anywhere in $1.0 \pm 1.00$ . At $B = 64$ :

\text{noise} = \frac{2.0}{\sqrt{64}} = \frac{2.0}{8} = 0.25

so a typical step lands in $1.0 \pm 0.25$ . Sixteen times the examples bought four times the steadiness, since $\sqrt{16} = 4$ . The diagram draws exactly this pair plus the full ladder $2.00$ , $1.00$ , $0.50$ , $0.25$ , $0.125$ for $B = 1, 4, 16, 64, 256$ .

Now count the steps. One pass over $N = 1024$ examples takes $N / B$ updates: $256$ steps at $B = 4$ but only $16$ at $B = 64$ . The same 16x jump that steadied each step 4x also cut the step count 16x. That mismatch, square-root steadiness against linear step loss, is the core tension of this page.

Why very large batches can test worse

Past some size, which shifts with the model and the data, big-batch runs keep reaching lower test accuracy than small-batch runs stopped at the same training loss. The standard account: small-batch noise acts as a mild regularizer, bouncing optimization out of sharp narrow dips and toward broad flat ones whose loss stays low on nearby unseen data. A huge batch averages that bounce away, so training settles wherever the slope points, sharp dip included.

Treat this as a real but qualitative effect. It shows up across many setups without a single number attached, so tune the batch size against a validation split instead of assuming the steadiest training generalizes best. The vocabulary for that comparison, train loss versus test behavior, is laid out in overfitting and underfitting.

Scaling the learning rate with the batch

A bigger batch gives a steadier gradient pointing roughly the same way, so practitioners raise the learning rate about in proportion when they raise $B$ : double the batch, roughly double the rate. This is the linear scaling rule, and it holds up to the sizes where the regularization effect above starts to bite.

One guardrail: the first few steps are fragile at high rates, so scaling pairs with a gradual ramp of the rate from near zero. That ramp is learning rate warmup, and when the GPU can't hold the batch you want, gradient accumulation builds the same average over several small steps first.

Code

The point is the check: recompute the noise ladder and the step counts from the worked example, and confirm each 4x jump in $B$ halves the noise.

1sigma: float = 2.02n_train: int = 10243
4for batch in [1, 4, 16, 64, 256]:5    noise: float = sigma / (batch ** 0.5)6    steps: int = n_train // batch7    print(f"B={batch:3d} noise={noise:.3f} steps={steps}")8# -> B=  1 noise=2.000 steps=10249# -> B=  4 noise=1.000 steps=25610# -> B= 16 noise=0.500 steps=6411# -> B= 64 noise=0.250 steps=1612# -> B=256 noise=0.125 steps=4

Each 4x step in batch size halves the noise ( $1/\sqrt{4} = 0.5$ ), matching the hand computation, while the step count quarters.

Watch Out For

Doubling the batch but keeping the old learning rate

Symptom: right after the change, training turns flat. Loss creeps instead of dropping, and nothing looks unstable, so the batch size never gets blamed. The steadier gradient at the old step size simply moves too cautiously.

Fix: scale the rate up roughly with the batch and add the ramp from learning rate warmup. Re-tune from there instead of trusting the old rate.

Filling the GPU with the biggest batch that fits

Symptom: training loss matches your best run but test accuracy lands clearly lower, and no amount of extra epochs closes it. The missing small-batch noise let training settle somewhere sharp.

Fix: treat batch size as tuned, not maximized. Search a few values against a validation split, and if memory forces small steps, hold the effective batch steady with gradient accumulation instead of shrinking it silently.

The Quick Version

Minibatch noise falls as $\sigma / \sqrt{B}$ , so doubling the batch cuts noise only about 1.4x.
Steps per pass fall as $N / B$ : on 1,024 examples, $B = 4$ gives 256 steps while $B = 64$ gives 16.
In our fixture ( $\sigma = 2.0$ ), 16x the examples steadies each step 4x, from $1.0 \pm 1.00$ to $1.0 \pm 0.25$ , while steps per pass fall 16x.
Very large batches can test worse than small ones at equal training loss, because small-batch noise regularizes toward flatter solutions.
When raising the batch, raise the learning rate roughly in proportion with a warmup ramp, and fake big batches with accumulation when memory runs short.