ZeRO & FSDP
Instead of every GPU holding a full model copy, each holds one shard and gathers the rest just in time, so cards pool memory instead of repeating it.
Why Does This Exist?
Copying the whole model stops working when the model stops fitting. Data parallelism keeps a full copy per card, which is ideal while it fits. A 7 billion parameter model in 16 bit needs 14GB for weights alone, yet training also needs gradients plus two Adam averages. That full copy is 84GB, beyond one 80GB card, and eight copies waste 672GB on repeats.
More cards do not fix repeats. You rent a row of GPUs and still cannot load, because each card repeats what its neighbour holds. Large runs hit this memory wall before they hit any compute limit, and the run never starts.
This page covers sharding that removes repeats. The full map lives in distributed training. Splits inside one layer live in tensor parallelism. Splits by depth live in pipeline parallelism.
You need one idea first. Data parallelism copies weights to every card, shards only data, then averages gradients with one all-reduce so all copies step together. When one copy no longer fits, you must shard the copy itself. That sharded copy is this page.
Think of It Like This
Eight cooks sharing one recipe book
One feast needs a hundred page recipe book. Giving every cook a full copy is simple while the book is thin. When the book grows thick, the shelf cannot hold eight copies.
Tear the book into eight chapters, one per cook. When a cook reaches a step from another chapter, they photocopy that page just in time, cook the step, then return the copy and keep only their own chapter. No cook ever holds the whole book, yet every dish still gets made.
Chapters are shards of weights, gradients, and optimizer states. Photocopies are all-gather messages. The extra walk to the copier is the communication price you pay for the shelf space you save.
Where it stops: cooks chat while the copier runs, while GPUs stall. Put the cooks in separate buildings with one slow copier and the kitchen slows to copier speed. Fast links are load-bearing here, not optional polish.
How It Actually Works
Call the parameter count , meaning trainable numbers. Weights in 16 bit need bytes, meaning 2 per number. Gradients in 16 bit need another bytes. Adam keeps two 32 bit averages per number, needing bytes. One training copy needs bytes before activations.
Stage 1 shards optimizer states
ZeRO stage 1 keeps full weights and gradients everywhere, but splits the optimizer averages across cards. Call the card count , meaning GPUs training together. Each card updates its own slice of weights, then broadcasts those fresh weights to the rest. Optimizer memory falls by , while weights and gradients stay full.
Stage 2 adds gradients
Stage 2 keeps stage 1 and also splits gradients. As soon as a gradient for one layer finishes, its card collects the pieces with a reduce-scatter, meaning each card keeps only its slice and drops the rest. Gradient memory falls by alongside optimizer memory. Weights stay full, so the forward pass still runs without extra fetches.
Stage 3 shards weights
Stage 3 splits everything, weights included. No card holds a full layer. Before a layer runs, cards gather its weights with an all-gather, meaning each card collects the missing slices just in time. They run the math, then drop what they fetched. Memory per card falls by . Traffic rises, because every layer now fetches before it computes in both directions.
FSDP is the native form of stage 3 thinking. It shards by layer, gathers one layer at a time, overlaps the fetch for the next layer with compute on the current one, then releases. Same arithmetic as ZeRO stage 3, with layer handling that suits large transformers.
Worked example: 7B across 8 cards
Take as 7 billion and as 8. Weights are GB, from times . Gradients are GB. Optimizer averages are GB, from times . One copy holds:
That is 84GB per card under plain data parallelism, past one 80GB card. Eight copies hold 672GB of repeats.
Stage 1 shards only the 56GB. Per card optimizer falls to , which is GB. Per card total is:
Stage 2 also shards gradients to , which is GB. Per card total is , which is GB.
Stage 3 shards all three to , which is GB:
Weights per shard are GB, so each layer fetch pulls the other GB over the link. That 12.25GB number is the trade in one line: 73.5GB saved per card against 12.25GB of weights moved per full pass. The diagram draws the 84GB stack beside the 10.5GB shard with that fetch arrow between them.
Combine with the single card levers when needed. Activation checkpointing cuts stored outputs, gradient accumulation grows the effective batch without more cards, and mixed precision training halves the math width. Sharding is the lever that pools cards.
Code
The point is the check: rebuild the 84GB copy and the 35GB and 10.5GB shards from counts, not quotes.
params: int = 7_000_000_000w_gb: float = params * 2 / 1e9g_gb: float = params * 2 / 1e9opt_gb: float = params * 8 / 1e9
total: float = w_gb + g_gb + opt_gbprint(f"{w_gb:.0f} {g_gb:.0f} {opt_gb:.0f} {total:.0f}") # -> 14 14 56 84print(f"{total / 8:.1f}") # -> 10.5print(f"{w_gb + g_gb + opt_gb / 8:.1f}") # -> 35.0The printout matches the hand arithmetic. Change 8 to 4 and watch the shard double to 21.0, which is the linear scaling claim in one edit.
Watch Out For
Sharding across a slow link and blaming the model
Stage 3 fetches weights for every layer, forward and backward. The symptom is exact: step time tracks the network, profiling shows collectives dominating, GPUs idle in lockstep, and adding cards slows the run. It looks like a model bug, so teams tune the model.
Fix: keep stage 3 and FSDP inside one fast fabric. Across racks, drop to stage 1 or 2 and add activation checkpointing instead. Measure the share of step time in collectives before renting more cards.
Sharding weights while activations still overflow
Sharding shrinks weights, gradients, and optimizer states. It touches no stored outputs. The symptom is an out-of-memory error that barely moves after stage 3, because a large batch of activations fills the card that weights just freed.
Fix: check what actually fills the card first. If activations dominate, add activation checkpointing or lower the per-card batch with gradient accumulation. Shard the state you hold, not the state you wish you held.
The Quick Version
- Data parallelism repeats a full 84GB copy per card for a 7B model, wasting 672GB across 8 cards.
- ZeRO stage 1 shards optimizer states to reach 35GB per card, stage 2 adds gradients to reach 22.75GB, stage 3 shards weights to reach 10.5GB.
- FSDP is the native sharded form that gathers each layer just in time and drops it right after.
- Sharding trades link traffic for memory, with 12.25GB of weights moved per pass against 73.5GB saved per card.
- Slow links stall sharded runs, so keep stage 3 inside fast fabric and use earlier stages elsewhere.
- Sharding leaves activations untouched, so checkpointing is the next lever when memory still overflows.