Skip to content
AI360XpertBeta

ZeRO & FSDP

Instead of every GPU holding a full model copy, each holds one shard and gathers the rest just in time, so cards pool memory instead of repeating it.

One 7B copy needs 84GB, while one ZeRO-3 shard of 8 needs 10.5GB because weights, gradients and optimizer states split across cards
One 7B copy needs 84GB, while one ZeRO-3 shard of 8 needs 10.5GB because weights, gradients and optimizer states split across cards

Why Does This Exist?

Copying the whole model stops working when the model stops fitting. Data parallelism keeps a full copy per card, which is ideal while it fits. A 7 billion parameter model in 16 bit needs 14GB for weights alone, yet training also needs gradients plus two Adam averages. That full copy is 84GB, beyond one 80GB card, and eight copies waste 672GB on repeats.

More cards do not fix repeats. You rent a row of GPUs and still cannot load, because each card repeats what its neighbour holds. Large runs hit this memory wall before they hit any compute limit, and the run never starts.

This page covers sharding that removes repeats. The full map lives in distributed training. Splits inside one layer live in tensor parallelism. Splits by depth live in pipeline parallelism.

You need one idea first. Data parallelism copies weights to every card, shards only data, then averages gradients with one all-reduce so all copies step together. When one copy no longer fits, you must shard the copy itself. That sharded copy is this page.

Think of It Like This

Eight cooks sharing one recipe book

One feast needs a hundred page recipe book. Giving every cook a full copy is simple while the book is thin. When the book grows thick, the shelf cannot hold eight copies.

Tear the book into eight chapters, one per cook. When a cook reaches a step from another chapter, they photocopy that page just in time, cook the step, then return the copy and keep only their own chapter. No cook ever holds the whole book, yet every dish still gets made.

Chapters are shards of weights, gradients, and optimizer states. Photocopies are all-gather messages. The extra walk to the copier is the communication price you pay for the shelf space you save.

Where it stops: cooks chat while the copier runs, while GPUs stall. Put the cooks in separate buildings with one slow copier and the kitchen slows to copier speed. Fast links are load-bearing here, not optional polish.

How It Actually Works

Call the parameter count PP, meaning trainable numbers. Weights in 16 bit need 2P2P bytes, meaning 2 per number. Gradients in 16 bit need another 2P2P bytes. Adam keeps two 32 bit averages per number, needing 8P8P bytes. One training copy needs 12P12P bytes before activations.

Stage 1 shards optimizer states

ZeRO stage 1 keeps full weights and gradients everywhere, but splits the optimizer averages across cards. Call the card count NN, meaning GPUs training together. Each card updates its own 1/N1/N slice of weights, then broadcasts those fresh weights to the rest. Optimizer memory falls by NN, while weights and gradients stay full.

Stage 2 adds gradients

Stage 2 keeps stage 1 and also splits gradients. As soon as a gradient for one layer finishes, its card collects the pieces with a reduce-scatter, meaning each card keeps only its slice and drops the rest. Gradient memory falls by NN alongside optimizer memory. Weights stay full, so the forward pass still runs without extra fetches.

Stage 3 shards weights

Stage 3 splits everything, weights included. No card holds a full layer. Before a layer runs, cards gather its weights with an all-gather, meaning each card collects the missing slices just in time. They run the math, then drop what they fetched. Memory per card falls by NN. Traffic rises, because every layer now fetches before it computes in both directions.

FSDP is the native form of stage 3 thinking. It shards by layer, gathers one layer at a time, overlaps the fetch for the next layer with compute on the current one, then releases. Same arithmetic as ZeRO stage 3, with layer handling that suits large transformers.

Worked example: 7B across 8 cards

Take PP as 7 billion and NN as 8. Weights are 1414GB, from 77 times 22. Gradients are 1414GB. Optimizer averages are 5656GB, from 77 times 88. One copy holds:

14+14+56=8414 + 14 + 56 = 84

That is 84GB per card under plain data parallelism, past one 80GB card. Eight copies hold 672GB of repeats.

Stage 1 shards only the 56GB. Per card optimizer falls to 56/856 / 8, which is 77GB. Per card total is:

14+14+7=3514 + 14 + 7 = 35

Stage 2 also shards gradients to 14/814 / 8, which is 1.751.75GB. Per card total is 14+1.75+714 + 1.75 + 7, which is 22.7522.75GB.

Stage 3 shards all three to 84/884 / 8, which is 10.510.5GB:

84/8=10.584 / 8 = 10.5

Weights per shard are 1.751.75GB, so each layer fetch pulls the other 12.2512.25GB over the link. That 12.25GB number is the trade in one line: 73.5GB saved per card against 12.25GB of weights moved per full pass. The diagram draws the 84GB stack beside the 10.5GB shard with that fetch arrow between them.

Combine with the single card levers when needed. Activation checkpointing cuts stored outputs, gradient accumulation grows the effective batch without more cards, and mixed precision training halves the math width. Sharding is the lever that pools cards.

Code

The point is the check: rebuild the 84GB copy and the 35GB and 10.5GB shards from counts, not quotes.

params: int = 7_000_000_000w_gb: float = params * 2 / 1e9g_gb: float = params * 2 / 1e9opt_gb: float = params * 8 / 1e9
total: float = w_gb + g_gb + opt_gbprint(f"{w_gb:.0f} {g_gb:.0f} {opt_gb:.0f} {total:.0f}")  # -> 14 14 56 84print(f"{total / 8:.1f}")  # -> 10.5print(f"{w_gb + g_gb + opt_gb / 8:.1f}")  # -> 35.0

The printout matches the hand arithmetic. Change 8 to 4 and watch the shard double to 21.0, which is the linear scaling claim in one edit.

Watch Out For

Sharding across a slow link and blaming the model

Stage 3 fetches weights for every layer, forward and backward. The symptom is exact: step time tracks the network, profiling shows collectives dominating, GPUs idle in lockstep, and adding cards slows the run. It looks like a model bug, so teams tune the model.

Fix: keep stage 3 and FSDP inside one fast fabric. Across racks, drop to stage 1 or 2 and add activation checkpointing instead. Measure the share of step time in collectives before renting more cards.

Sharding weights while activations still overflow

Sharding shrinks weights, gradients, and optimizer states. It touches no stored outputs. The symptom is an out-of-memory error that barely moves after stage 3, because a large batch of activations fills the card that weights just freed.

Fix: check what actually fills the card first. If activations dominate, add activation checkpointing or lower the per-card batch with gradient accumulation. Shard the state you hold, not the state you wish you held.

The Quick Version

  • Data parallelism repeats a full 84GB copy per card for a 7B model, wasting 672GB across 8 cards.
  • ZeRO stage 1 shards optimizer states to reach 35GB per card, stage 2 adds gradients to reach 22.75GB, stage 3 shards weights to reach 10.5GB.
  • FSDP is the native sharded form that gathers each layer just in time and drops it right after.
  • Sharding trades link traffic for memory, with 12.25GB of weights moved per pass against 73.5GB saved per card.
  • Slow links stall sharded runs, so keep stage 3 inside fast fabric and use earlier stages elsewhere.
  • Sharding leaves activations untouched, so checkpointing is the next lever when memory still overflows.