Skip to content
AI360XpertBeta

Activation Checkpointing

Training usually keeps every layer's output in memory for the backward pass. Checkpointing keeps only a few of them and rebuilds the rest on demand, trading about a third more compute for far less memory.

Keeping every fourth layer output and rebuilding the rest during backprop cuts memory from 128 MB to 64 MB in a 16-layer network
Keeping every fourth layer output and rebuilding the rest during backprop cuts memory from 128 MB to 64 MB in a 16-layer network

Why Does This Exist?

Training runs out of memory long before it runs out of ideas. And the culprit is rarely the weights. For a deep network with a serious batch size, the stored activations (every layer's output, saved for the backward pass) dwarf everything else in VRAM. Double the batch and you double that storage; add layers and it grows again. Sooner or later you meet the out-of-memory error, and a model that can't fit can't train.

The obvious fix is a smaller batch. That fits, but it wastes the GPU: tiny batches mean noisy gradients and idle arithmetic units. You want the big batch without the big storage bill.

That's the deal this page covers. Activation checkpointing (also called gradient checkpointing) keeps a few layer outputs and rebuilds the rest by re-running short stretches of the forward pass exactly when the backward pass needs them. Compute is spent to buy back memory. Its closest sibling is gradient accumulation, which attacks the same out-of-memory pain from the batch side; the mechanism here lives inside backpropagation itself.

The one idea you need first: backprop needs those stored values because each layer's gradient is computed from its own output, by the chain rule. Skip the storing and the backward pass has nothing to work with. The full treatment is in backpropagation.

Think of It Like This

Trail notes instead of trail photos

You are hiking a 16-mile trail and you'll need to describe every mile on the way back. One option is photographing every single step: a complete record and an enormous camera roll. The other is jotting a short note every 4 miles and re-walking only the stretch you're describing when the time comes. Your pack stays light because notes are cheap; you pay with your legs instead.

Checkpointing is the second hiker. The kept layer outputs are the notes, the rebuilt segments are the re-walked miles, and the extra forward passes are the extra steps. The analogy stops where determinism starts: re-walking a trail never reproduces it exactly, while re-running a layer reproduces its output bit for bit, which is the whole reason the trick is legal. Put anything non-deterministic inside a rebuilt stretch and that guarantee breaks, as the pitfalls below show.

How It Actually Works

Why backprop hoards memory

Say your network has NN layers. During the forward pass, layer ii turns its input xix_i into an output aia_i (the activation: the layer's computed result). Backprop needs aia_i again to form the gradient at layer ii, by the chain rule, so the framework holds every aia_i from the moment it's computed until the backward pass consumes it. If one activation costs aa bytes, the bill is N×aN \times a, held for the entire backward pass. Nothing about this is wasteful in spirit: those values are genuinely needed. The waste is timing. Each one sits in memory while every later layer takes its turn.

Store few, rebuild the rest

Pick a segment length kk (you'll see why it wants to sit near N\sqrt{N}). During the forward pass, keep only every kk-th activation: layers kk, 2k2k, 3k3k, and so on. These are the checkpoints. Everything between them is deleted as soon as the next checkpoint is safely stored.

When the backward pass reaches a segment, say the stretch between checkpoint kk and checkpoint 2k2k, it re-runs the forward pass from checkpoint kk up through layer 2k2k, holding that segment's kk live activations just long enough to compute its gradients, then drops them and moves to the earlier segment. One extra forward run per segment. Total extra work: about one full forward pass.

Peak memory is now the checkpoints plus one live segment: N/k+kN/k + k activations. That sum is smallest when the two terms match, k=Nk = \sqrt{N}, giving roughly 2N2\sqrt{N} instead of NN:

stored activations≈Nk+k      (smallest when k=N)\text{stored activations} \approx \frac{N}{k} + k \;\;\; \left( \text{smallest when } k = \sqrt{N} \right)

On cost: a forward pass does about one heavy multiply per layer where the backward pass does about two, so one extra forward pass adds roughly a third to the step. Memory falls from linear to square-root scaling; time grows by a fraction. And when the saving lets you raise the batch size (see batch size effects), the bigger batch often repays that fraction with interest.

Worked example: 16 layers, checkpoints every 4

Take N=16N = 16 layers, one activation costs a=8a = 8 MB, checkpoints at every k=4k = 4th layer.

Baseline: hold all 16 outputs. 16×8=12816 \times 8 = 128 MB.

Checkpointed forward pass: keep layers 4, 8, 12, 16. That's 16/4=416/4 = 4 checkpoints at 4×8=324 \times 8 = 32 MB.

Backward through the last segment (layers 13 to 16): re-run forward from the layer-12 checkpoint, holding 4 live activations (4×8=324 \times 8 = 32 MB) alongside the 4 checkpoints (3232 MB). Peak: 32+32=6432 + 32 = 64 MB. Half the memory. Every segment behaves the same, so the whole backward pass peaks at 64 MB against the baseline's 128 MB, for one extra forward pass worth of compute.

Check the general formula: N/k+k=16/4+4=8N/k + k = 16/4 + 4 = 8 layer-worths, and 8×8=648 \times 8 = 64 MB. It agrees.

When it doesn't help

Checkpointing shrinks activation storage and nothing else. If your out-of-memory error comes from the weights, the optimizer state, or a single giant layer that dwarfs the rest, recomputing changes nothing: there's no crowd of stored activations to thin out. Fix the footprint problem you actually have. Optimizer-heavy setups pair this with other tools instead, which is why checkpointing usually appears inside a larger scaling plan like distributed training.

One useful refinement: checkpoint at coarse boundaries, such as whole transformer blocks, and let cheap elementwise steps fall inside rebuilt segments rather than occupying checkpoints. The activation functions themselves cost almost nothing to recompute, so storing their outputs instead of rebuilding them is pure waste.

Code

The snippet below prices the trade for the worked example: peak activation memory with and without checkpointing, in megabytes.

def peak_memory(num_layers: int, every: int, per_layer_mb: float) -> float:    checkpoints = num_layers // every  # one kept output per segment    live_segment = every  # one rebuilt segment held during backprop    return (checkpoints + live_segment) * per_layer_mb

N = 16print(peak_memory(N, 4, 8.0))  # -> 64.0print(N * 8.0)                 # -> 128.0print(128.0 / 64.0)            # -> 2.0

Those are the numbers the worked example derived by hand: 64 MB against 128 MB, a factor of 2.0 from one extra forward pass. The // is exact here because 4 divides 16; with a remainder the final segment is shorter, which only lowers the peak.

Watch Out For

Rebuilding a segment that isn't deterministic

The trick is only legal because re-running a layer reproduces its output exactly. Anything inside a checkpointed segment that rolls the dice differently on the second run (unseeded dropout masks, noise added in place, a branch on shifting input statistics) makes the backward pass differentiate a different function from the one the forward pass evaluated.

The symptom is maddening: you enable checkpointing, change nothing else, and the loss trajectory shifts, or gradients stop matching a no-checkpointing run beyond tolerance. It looks like a modeling bug, so people debug the model.

Keep randomness outside checkpointed segments or seed it so both runs draw identically, and never modify a checkpointed input in place between the two runs. Then confirm the fix the cheap way: run a few steps with and without checkpointing and compare losses exactly before training for real.

Checkpointing the wrong memory problem

Checkpointing thins stored activations. It doesn't shrink weights, optimizer state, or one enormous layer. If those dominate your footprint, you'll pay the extra forward pass and still run out of memory, then conclude the technique "doesn't work."

The symptom is an out-of-memory error that barely moves when you add checkpoints. Before reaching for this, check what actually fills the card: if activations dominate, checkpoint; if the optimizer state does, look at accumulation or sharding strategies first (see gradient accumulation). And checkpoint whole blocks, not every layer: wrapping each layer separately maximizes bookkeeping while barely lowering the peak.

The Quick Version

  • Backprop normally holds every layer's output until the backward pass consumes it, and that storage is usually what overflows the GPU.
  • Checkpointing keeps every kk-th output and deletes the rest, rebuilding each segment with a short extra forward run when the backward pass arrives.
  • Stored memory drops from NN to about N/k+kN/k + k layer outputs, smallest near k=Nk = \sqrt{N}: square-root scaling instead of linear.
  • The price is about one extra forward pass per step, roughly a third more compute, often repaid by the larger batch the savings allow.
  • It only shrinks activation storage: weights, optimizer state, and single giant layers are untouched.
  • Rebuilt segments must be deterministic: unseeded randomness inside one silently corrupts gradients.