Activation Checkpointing
Training usually keeps every layer's output in memory for the backward pass. Checkpointing keeps only a few of them and rebuilds the rest on demand, trading about a third more compute for far less memory.
Why Does This Exist?
Training runs out of memory long before it runs out of ideas. And the culprit is rarely the weights. For a deep network with a serious batch size, the stored activations (every layer's output, saved for the backward pass) dwarf everything else in VRAM. Double the batch and you double that storage; add layers and it grows again. Sooner or later you meet the out-of-memory error, and a model that can't fit can't train.
The obvious fix is a smaller batch. That fits, but it wastes the GPU: tiny batches mean noisy gradients and idle arithmetic units. You want the big batch without the big storage bill.
That's the deal this page covers. Activation checkpointing (also called gradient checkpointing) keeps a few layer outputs and rebuilds the rest by re-running short stretches of the forward pass exactly when the backward pass needs them. Compute is spent to buy back memory. Its closest sibling is gradient accumulation, which attacks the same out-of-memory pain from the batch side; the mechanism here lives inside backpropagation itself.
The one idea you need first: backprop needs those stored values because each layer's gradient is computed from its own output, by the chain rule. Skip the storing and the backward pass has nothing to work with. The full treatment is in backpropagation.
Think of It Like This
Trail notes instead of trail photos
You are hiking a 16-mile trail and you'll need to describe every mile on the way back. One option is photographing every single step: a complete record and an enormous camera roll. The other is jotting a short note every 4 miles and re-walking only the stretch you're describing when the time comes. Your pack stays light because notes are cheap; you pay with your legs instead.
Checkpointing is the second hiker. The kept layer outputs are the notes, the rebuilt segments are the re-walked miles, and the extra forward passes are the extra steps. The analogy stops where determinism starts: re-walking a trail never reproduces it exactly, while re-running a layer reproduces its output bit for bit, which is the whole reason the trick is legal. Put anything non-deterministic inside a rebuilt stretch and that guarantee breaks, as the pitfalls below show.
How It Actually Works
Why backprop hoards memory
Say your network has layers. During the forward pass, layer turns its input into an output (the activation: the layer's computed result). Backprop needs again to form the gradient at layer , by the chain rule, so the framework holds every from the moment it's computed until the backward pass consumes it. If one activation costs bytes, the bill is , held for the entire backward pass. Nothing about this is wasteful in spirit: those values are genuinely needed. The waste is timing. Each one sits in memory while every later layer takes its turn.
Store few, rebuild the rest
Pick a segment length (you'll see why it wants to sit near ). During the forward pass, keep only every -th activation: layers , , , and so on. These are the checkpoints. Everything between them is deleted as soon as the next checkpoint is safely stored.
When the backward pass reaches a segment, say the stretch between checkpoint and checkpoint , it re-runs the forward pass from checkpoint up through layer , holding that segment's live activations just long enough to compute its gradients, then drops them and moves to the earlier segment. One extra forward run per segment. Total extra work: about one full forward pass.
Peak memory is now the checkpoints plus one live segment: activations. That sum is smallest when the two terms match, , giving roughly instead of :
On cost: a forward pass does about one heavy multiply per layer where the backward pass does about two, so one extra forward pass adds roughly a third to the step. Memory falls from linear to square-root scaling; time grows by a fraction. And when the saving lets you raise the batch size (see batch size effects), the bigger batch often repays that fraction with interest.
Worked example: 16 layers, checkpoints every 4
Take layers, one activation costs MB, checkpoints at every th layer.
Baseline: hold all 16 outputs. MB.
Checkpointed forward pass: keep layers 4, 8, 12, 16. That's checkpoints at MB.
Backward through the last segment (layers 13 to 16): re-run forward from the layer-12 checkpoint, holding 4 live activations ( MB) alongside the 4 checkpoints ( MB). Peak: MB. Half the memory. Every segment behaves the same, so the whole backward pass peaks at 64 MB against the baseline's 128 MB, for one extra forward pass worth of compute.
Check the general formula: layer-worths, and MB. It agrees.
When it doesn't help
Checkpointing shrinks activation storage and nothing else. If your out-of-memory error comes from the weights, the optimizer state, or a single giant layer that dwarfs the rest, recomputing changes nothing: there's no crowd of stored activations to thin out. Fix the footprint problem you actually have. Optimizer-heavy setups pair this with other tools instead, which is why checkpointing usually appears inside a larger scaling plan like distributed training.
One useful refinement: checkpoint at coarse boundaries, such as whole transformer blocks, and let cheap elementwise steps fall inside rebuilt segments rather than occupying checkpoints. The activation functions themselves cost almost nothing to recompute, so storing their outputs instead of rebuilding them is pure waste.
Code
The snippet below prices the trade for the worked example: peak activation memory with and without checkpointing, in megabytes.
def peak_memory(num_layers: int, every: int, per_layer_mb: float) -> float: checkpoints = num_layers // every # one kept output per segment live_segment = every # one rebuilt segment held during backprop return (checkpoints + live_segment) * per_layer_mb
N = 16print(peak_memory(N, 4, 8.0)) # -> 64.0print(N * 8.0) # -> 128.0print(128.0 / 64.0) # -> 2.0Those are the numbers the worked example derived by hand: 64 MB against 128 MB, a factor of 2.0 from one extra forward pass. The // is exact here because 4 divides 16; with a remainder the final segment is shorter, which only lowers the peak.
Watch Out For
Rebuilding a segment that isn't deterministic
The trick is only legal because re-running a layer reproduces its output exactly. Anything inside a checkpointed segment that rolls the dice differently on the second run (unseeded dropout masks, noise added in place, a branch on shifting input statistics) makes the backward pass differentiate a different function from the one the forward pass evaluated.
The symptom is maddening: you enable checkpointing, change nothing else, and the loss trajectory shifts, or gradients stop matching a no-checkpointing run beyond tolerance. It looks like a modeling bug, so people debug the model.
Keep randomness outside checkpointed segments or seed it so both runs draw identically, and never modify a checkpointed input in place between the two runs. Then confirm the fix the cheap way: run a few steps with and without checkpointing and compare losses exactly before training for real.
Checkpointing the wrong memory problem
Checkpointing thins stored activations. It doesn't shrink weights, optimizer state, or one enormous layer. If those dominate your footprint, you'll pay the extra forward pass and still run out of memory, then conclude the technique "doesn't work."
The symptom is an out-of-memory error that barely moves when you add checkpoints. Before reaching for this, check what actually fills the card: if activations dominate, checkpoint; if the optimizer state does, look at accumulation or sharding strategies first (see gradient accumulation). And checkpoint whole blocks, not every layer: wrapping each layer separately maximizes bookkeeping while barely lowering the peak.
The Quick Version
- Backprop normally holds every layer's output until the backward pass consumes it, and that storage is usually what overflows the GPU.
- Checkpointing keeps every -th output and deletes the rest, rebuilding each segment with a short extra forward run when the backward pass arrives.
- Stored memory drops from to about layer outputs, smallest near : square-root scaling instead of linear.
- The price is about one extra forward pass per step, roughly a third more compute, often repaid by the larger batch the savings allow.
- It only shrinks activation storage: weights, optimizer state, and single giant layers are untouched.
- Rebuilt segments must be deterministic: unseeded randomness inside one silently corrupts gradients.