Core ML

Batch Size Effects

The batch size sets how noisy each gradient estimate is, trading off against how many steps fit in a compute budget and how well the model ends up generalizing.

Gradient noise falls as the square root of batch size, so doubling the batch buys a shrinking return, while steps per epoch fall in direct proportion to batch size

Why Does This Exist?

Every minibatch gradient is an estimate of the gradient over the entire training set, computed from a small random sample of it rather than all of it. That estimate carries noise, and the batch size is the one number that controls exactly how much. It's tempting to treat batch size as a resource-allocation knob — "how many examples fit on the GPU" — and stop there, but it also changes how noisy training is, how many optimizer steps a fixed amount of data produces, and, more surprisingly, how well the final model generalizes to data it hasn't seen. None of those three effects move together in a simple way, which is why batch size is a real tuning decision and not just a memory constraint.

Think of It Like This

Polling a small group versus a large one

Estimating the average height of a city's population by measuring 10 random people gives a noisy, easily-skewed estimate — one unusually tall person shifts the average a lot. Measure 1,000 random people instead and the average settles down: individual outliers matter far less, and the estimate sits much closer to the city's true average.

But going from 10 people to 1,000 costs 100 times the effort, and going from 1,000 to 100,000 barely improves the estimate further — you've already captured most of the population's real variation. Gradient noise behaves the same way: more examples per batch means a steadier estimate, with rapidly diminishing returns the larger the batch already is.

How It Actually Works

Noise falls as the square root of batch size, not in proportion to it

A minibatch gradient's estimation error — its standard deviation around the true full-dataset gradient — scales as $\sigma / \sqrt{B}$ , where $\sigma$ is the per-example gradient variability and $B$ is the batch size. Doubling the batch size only shrinks the noise by a factor of $\sqrt{2} \approx 1.41$ , not by half. Going from batch size 8 to 512, a 64x increase, cuts noise by only $\sqrt{64} = 8$ x. That square-root relationship is the whole reason batch size has diminishing returns rather than a clean linear tradeoff.

Steps per epoch falls in direct proportion, not as a square root

One full pass over a fixed-size training set takes $N/B$ optimizer steps, where $N$ is the dataset size — this shrinks linearly with batch size, far faster than noise falls. Doubling the batch size halves the number of steps per epoch outright. Combined with the square-root noise result above, this is the core tension: a bigger batch buys a modest, shrinking reduction in per-step noise while cutting the number of steps taken by a lot more.

The generalization gap

Very large batch sizes, past a point that depends on the model and dataset, are repeatedly observed to reach a worse final test accuracy than a smaller batch size trained for the same number of epochs — even when both converge to similarly low training loss. The gradient noise a small batch produces is now understood to act as a mild regularizer, nudging optimization away from sharp minima that generalize poorly and toward flatter ones that generalize better. Push the batch size up far enough and that noise all but disappears, and some of that implicit regularization disappears with it.

The linear scaling rule

Practitioners scaling up batch size to use more parallel hardware commonly scale the learning rate up by roughly the same factor, combined with warmup to avoid instability in the first steps. The reasoning follows directly from the noise result above: a larger batch produces a less noisy but still similarly-shaped gradient, so a proportionally larger step continues to make sense, up to the point where the generalization gap above starts to bite.

Show Me the Code

Simulated per-example gradient noise, showing the estimate's standard deviation actually shrink at a $1/\sqrt{B}$ rate rather than $1/B$ .

1import numpy as np2
3rng = np.random.default_rng(0)4true_grad, per_example_std = 1.0, 2.05
6for batch_size in [8, 32, 128, 512]:7    samples = rng.normal(true_grad, per_example_std, size=(2000, batch_size)).mean(axis=1)8    print(f"batch {batch_size:4d}  empirical std {samples.std():.4f}")9# -> batch    8  empirical std 0.702410# -> batch   32  empirical std 0.354611# -> batch  128  empirical std 0.175412# -> batch  512  empirical std 0.0881

Each 4x jump in batch size roughly halves the noise, matching $1/\sqrt{4}=0.5$ — not the 4x reduction a naive "more data averages out more noise" intuition might predict.

Watch Out For

Increasing batch size without adjusting the learning rate

A learning rate tuned for batch size 32 applied unchanged at batch size 512 is now working with a far less noisy gradient estimate at the same step size — training often looks flat and painfully slow, not unstable, which makes the actual cause easy to miss. The linear scaling rule above exists specifically to catch this.

Chasing the largest batch size the hardware allows, by default

Maximizing batch size to maximize hardware throughput is a reasonable instinct for speed per step, but it can quietly trade away the generalization benefit smaller-batch noise was providing. The right batch size is a tuned choice balancing throughput, stability, and final model quality — not simply "as large as fits."

The Quick Version

A minibatch gradient's noise falls as $1/\sqrt{B}$ — diminishing returns as batch size grows.
Steps per epoch fall as $1/B$ — a much steeper drop than the noise reduction.
Very large batches can reach worse test performance than smaller ones at the same training loss, because small-batch noise acts as a mild regularizer.
The linear scaling rule raises the learning rate roughly in proportion to batch size, paired with warmup, when scaling up.
Batch size is a real tuning decision balancing throughput, training stability, and generalization — not purely a memory constraint.