Gradient Accumulation
Summing the gradients from several small mini-batches before running one optimizer step, simulating a larger batch size on limited GPU memory.
Several small forward-backward passes run one after another, each adding its gradient into a shared buffer instead of triggering an update. Only after the last micro-batch in the group does the accumulated gradient get divided by the count and handed to the optimizer — arithmetically identical to averaging over one real batch of that combined size, just computed in pieces to fit in memory.
The trade is wall-clock time for memory, and batch normalization doesn't accumulate cleanly across micro-batches the way layer or group normalization does.