Batch Size Effects
How the number of examples averaged into one gradient estimate changes training noise, the largest stable learning rate, and final generalization.
Think of It Like This
Like surveying a handful of noisy voters to guess an election versus polling an entire stadium: small groups take wild swings, while large crowds move steadily.
A minibatch gradient's noise falls as one over the square root of the batch size, while the number of steps per epoch falls in direct proportion to it — a much steeper drop. Doubling the batch only cuts noise by about 1.4x, but it also halves the number of steps a fixed dataset produces.
Past a certain size, very large batches are repeatedly observed to reach worse test performance than smaller ones trained to the same training loss — the noise a small batch introduces acts as a mild regularizer that a large, low-noise batch loses.