DeepSpeed Training

DeepSpeed shards optimizer memory and offloads smartly, so billion-parameter vision models train on hardware teams already own.

Fifty six gigabytes of Adam state shrink to seven per GPU when ZeRO stage 3 shards parameters, gradients, and optimizer states across eight cards

Why Does This Exist?

A 7 billion parameter model in 16-bit needs about 14 GB for weights alone. Adam adds two full-precision states per parameter plus a master copy, roughly 56 GB more. No single GPU holds that, and naive data parallel copies it everywhere, multiplying the waste.

DeepSpeed removes the copies. Its ZeRO stages shard optimizer states, then gradients, then parameters across workers, while offload moves cold tensors to CPU or NVMe. Data parallelism is the baseline it fixes, efficient pretraining sets the budget, and inference optimization handles the model after training.

Think of It Like This

A shared workshop instead of full toolboxes

Eight carpenters each owning every tool need eight full sets. A shared workshop keeps one set on labelled shelves: whoever needs the lathe walks over, uses it, and returns it.

Parameters are the tools and GPUs are carpenters. Sharding keeps one copy spread across shelves instead of eight full sets. The analogy stops at speed: walking takes time, and DeepSpeed pays it in communication between steps.

How It Actually Works

ZeRO stage 1 shards optimizer states only. Stage 2 adds gradients. Stage 3 adds parameters, so each GPU holds one eighth of everything and gathers what it needs per layer just in time. Offload extends the shelf to CPU RAM or NVMe for tensors not needed this step, and mixed precision keeps compute in 16-bit while master weights stay in 32-bit. Communication grows as sharding deepens, so stage choice trades memory for network traffic.

Worked example

Seven billion parameters: weights $7 \times 2 = 14$ GB in 16-bit. Adam states: two moments at 4 bytes each plus master copy at 4 bytes gives $12$ bytes per parameter, $84$ GB total, of which about $56$ GB is the extra moment state teams quote. ZeRO-3 over $8$ GPUs cuts resident state to roughly $7$ GB each plus gathered slices in flight. The model that overflowed one $80$ GB card now fits with room for activations.

Code

Memory math before reaching for hardware.

1def adam_shard_gb(params_b: float = 7.0, gpus: int = 8) -> tuple[float, float]:2    """Total Adam extra state and per-GPU share under ZeRO-3."""3    total = params_b * 8.0  # two fp32 moments, 4 bytes each per param4    return total, total / gpus5
6
7print(adam_shard_gb())  # -> (56.0, 7.0) GB total and per GPU

Watch Out For

Offloading everything to save GPUs

Pushing all states to NVMe frees cards but starves the step: every layer waits on drive reads and throughput falls off a cliff. The symptom is low GPU use beside a hot storage queue.

Offload optimizer states first, keep hot parameters resident. Measure step time after each offload knob, not just peak memory.

The Quick Version

DeepSpeed shards optimizer states, gradients, and parameters across workers through ZeRO stages.
Offload extends capacity to CPU and NVMe at the cost of step time.
Worked numbers: $56$ GB of Adam state becomes about $7$ GB per GPU over $8$ cards.
Mixed precision keeps compute cheap while master weights stay precise.
Offload cold states first and time real steps, not just memory gauges.