Gradient Accumulation

Run several small forward-backward passes and sum their gradients before one optimizer step, so limited GPU memory can still train as if it saw one large batch.

Four micro-batches of 16 with means 0.8, 0.4, 0.6 and 0.2 sum to 2.0 and average to 0.5, so one optimizer step on 0.5 matches a real batch of 64

Why Does This Exist?

Weights, saved activations, and one batch of inputs must fit in GPU memory together. Batch size effects explains why a target batch size is right for stability or throughput, but the size the recipe calls for and the size that fits are often two different numbers. Buying a bigger card is one answer. Accumulation trains at the target size on the card you already own.

The prerequisite is one sentence. The gradient of a batch is the average of its per-example gradients, and the optimizer steps on that average. If you can rebuild the same average in pieces, you can take the same step. Gradient descent is that step, and this page is how to feed it a large-batch average without holding a large batch.

Think of It Like This

Filling a bucket with a cup

A recipe calls for one full bucket added to a mix at once, but you only own a small cup. You carry cup after cup, keep a running count, and stir only once the total equals a full bucket. Four cups of 16 spoonfuls make the 64 the recipe wanted.

Each cup is one micro-batch forward-backward pass, the running count is the gradient buffer, and the single stir is the one optimizer step after the fourth cup.

Where it stops: water adds perfectly and buckets never overflow mid-recipe. Gradients add perfectly only for the mean, while layers that read whole-batch statistics such as batch normalization still see four small groups instead of one group of 64. Cups also cost trips, so you trade time for memory rather than getting the bucket for free.

How It Actually Works

The accumulation loop

For $K$ micro-batches that stand in for one large batch, run forward and backward on each micro-batch, add each gradient into the same buffer without clearing it, then step once:

g_{eff} = \frac{1}{K}\sum_{i=1}^{K} g_i

Here $g_i$ is the mean gradient of micro-batch $i$ , meaning its per-example gradients averaged, $K$ is the accumulation count, and $g_{eff}$ is the effective gradient handed to the optimizer. Only one micro-batch of activations lives in memory at a time. The buffer holding $g_{eff}$ costs one copy of the parameters, which is small next to activations for a large batch.

Worked example: four means that average to 0.50

Take micro-batch size 16 and $K = 4$ , so the effective batch is $16 \times 4 = 64$ . Say the four micro-batch mean gradients on one weight are:

[0.80, 0.40, 0.60, 0.20]

Add: $0.80 + 0.40 + 0.60 + 0.20 = 2.00$ . Divide by $K$ :

g_{eff} = 2.00 / 4 = 0.50

A real batch of 64 with the same 64 examples averages to the same $0.50$ by the same addition in a different order. The diagram draws these four contributions flowing into one summed buffer and one step. Scaling the learning rate is unchanged because the optimizer sees the same averaged magnitude it would have seen.

What changes and what does not

Compute is not saved. Four passes cost about four times one large pass that fit, so this buys memory at the price of wall-clock time. Anything that reads the whole batch at once still differs, mainly batch normalization, whose per-micro-batch means and variances are not the full-batch statistics. That is why accumulation pairs with layer or group normalization, and with mixed precision training, the other lever on the same memory limit.

Code

The point is the check: rebuild the 0.50 above from four micro-batch means and confirm it matches one direct average.

1micro_means: list[float] = [0.80, 0.40, 0.60, 0.20]2k: int = len(micro_means)3
4effective: float = sum(micro_means) / k5direct: float = sum([0.80, 0.40, 0.60, 0.20]) / 46
7print(f"{effective:.2f} {direct:.2f} {effective == direct}")  # -> 0.50 0.50 True8print(f"effective batch {16 * 4}")  # -> effective batch 64

The printout matches the hand addition: both paths give 0.50, and 16 times 4 gives the effective batch of 64. Change one entry from 0.20 to 1.00 and both paths move together to 0.70, which is the exactness claim in one line.

Watch Out For

Summing without dividing by K

Symptom: training destabilizes right after turning accumulation on, with loss spikes that look like a too-large learning rate even though the rate never changed. Gradients read about $K$ times too large.

Fix: divide the buffer by $K$ before stepping, or equivalently scale each micro-batch loss by $1/K$ before backward. Confirm with the snippet above that $K = 4$ gives 0.50, not 2.00.

Keeping batch normalization across micro-batches

Symptom: accumulated runs underperform the same effective batch run on bigger hardware, with noisy validation curves that do not close by tuning the rate. Per-micro-batch means stand in for a full-batch mean they do not equal.

Fix: use layer or group normalization where the recipe allows, or accumulate only where normalization statistics do not cross the micro-batch boundary. Log which normalization the model uses before blaming the optimizer.

The Quick Version

Gradient accumulation runs $K$ small forward-backward passes, sums their gradients, divides by $K$ , and steps once.
Means 0.80, 0.40, 0.60, and 0.20 sum to 2.00 and average to 0.50, matching a real batch of 64 built as 16 times 4.
The averaged gradient is arithmetically identical to the large-batch average, not an approximation of it.
The trade is time for memory: $K$ sequential passes cost about $K$ times one large pass that fit.
Batch normalization does not accumulate cleanly, so layer or group normalization is the common pairing.