Skip to content
AI360XpertBeta

Distributed Training

One GPU cannot hold a 70-billion parameter model, so split the weights, the layers, or the data itself across many GPUs that train together as one machine.

A 70 billion parameter model needs 140GB in 16-bit against an 80GB chip, so data parallelism copies the model and splits batches while pipeline and tensor parallelism split layers and matrices
A 70 billion parameter model needs 140GB in 16-bit against an 80GB chip, so data parallelism copies the model and splits batches while pipeline and tensor parallelism split layers and matrices

Why Does This Exist?

Count the bytes. A 70-billion parameter model at 16-bit precision needs 2 bytes per parameter, which is 140GB just to hold the weights. The largest widely used training chip holds 80GB. The model does not fit, and that is before gradients, optimizer states, and activations: full Adam states in 32-bit need roughly 12 bytes per parameter, another 840GB for the same model. Fitting is impossible on one chip, and even a model that fits would train for years alone.

So training became an infrastructure problem: stitch many GPUs together over fast links so they behave as one computer, with each holding a slice of the work. Three slicing strategies cover nearly every large run, often combined in one job.

This page covers the three strategies and the arithmetic that forces them. Each strategy has its own page next: data parallelism, tensor parallelism, and pipeline parallelism. The single-GPU fallback that stretches memory without more chips is gradient accumulation.

Think of It Like This

One book, one hundred translators, one day

A 10,000-page manual must be translated overnight. One translator cannot finish, so you hire a hundred and split the work three ways at once.

First, print one dictionary per translator and hand each a different chapter: everyone works at full speed on their own pages, then you reconcile the terminology at the end. Second, arrange others in a line where the first sketches each section, the second writes prose from the sketch, and the third polishes: each page flows down the line, though the end of the line idles while the first pages arrive. Third, for one monster sentence nobody can swallow whole, two translators split it word by word and merge each line before moving on, talking constantly.

The mapping: the dictionaries are model copies, the chapters are data batches, the line is the layer stack, the monster sentence is one giant matrix, and the reconciling is gradient synchronization. Where it stops: translators tolerate slow mail, while GPUs need ultra-fast links for the constant chatter, and the idle time at the ends of the line, the pipeline bubble, has no human counterpart worth keeping.

How It Actually Works

The memory wall, in numbers

Let PP be the parameter count. At 16-bit precision each parameter takes 2 bytes, so weights alone need 2P2P bytes. For P=70P = 70 billion:

70×109×2=140×109 bytes=140 GB70 \times 10^9 \times 2 = 140 \times 10^9\ \text{bytes} = 140\,\text{GB}

Against an 80GB chip that is 140/80=1.75140 / 80 = 1.75 chips for weights alone, so at least two, before anything else. Full Adam keeps a 32-bit master copy plus two 32-bit running averages per parameter, 12 bytes each round:

70×109×12=840 GB70 \times 10^9 \times 12 = 840\,\text{GB}

No single chip is close. Something must be split, and the three strategies below are the complete menu of what can be split: the data, the layer stack, or the matrices themselves.

Data parallelism: copy the model, split the batches

Every GPU holds a full model copy and processes a different slice of the batch. Each computes its own gradients then all GPUs average them, an all-reduce, so every copy updates identically. The constraint is absolute: the whole model must fit on each chip, which the 70-billion case above already violates.

The communication cost is predictable. In a ring all-reduce over NN GPUs, each GPU moves about 2(N−1)/N2(N-1)/N times the gradient size. With N=4N = 4 that factor is 2×3/4=1.52 \times 3/4 = 1.5. A 7-billion parameter model in 16-bit has 14GB of gradients, so each GPU moves 14×1.5=2114 \times 1.5 = 21GB per step. That is the diagram's center panel: four identical boxes, four different batches, one synchronization arrow worth 21GB.

Pipeline parallelism: split the layer stack

When the model exceeds one chip, slice it by depth: layers 1 to 10 on GPU 1, 11 to 20 on GPU 2, and so on. Activations flow down the line. The price is the pipeline bubble: early GPUs idle while later ones wait for first inputs, and late ones idle while early ones finish. Micro-batching, chopping each batch into smaller pieces that chase each other down the line, shrinks but never removes it.

Tensor parallelism: split the matrices themselves

When a single layer is too big, cut its matrix multiplications across GPUs: each computes part of the same multiplication and the partial results merge immediately. This demands the fastest links available, because the GPUs must talk inside every single calculation rather than once per step.

Worked example: placing the 70-billion model

Weights need 140GB and the chip holds 80GB, so weights alone span ⌈140/80⌉=2\lceil 140/80 \rceil = 2 chips at minimum. Add gradients at 16-bit, another 140GB, and the running total is 280GB, or ⌈280/80⌉=4\lceil 280/80 \rceil = 4 chips before optimizer states. Full Adam states add 840GB, taking the total to 1120GB, or ⌈1120/80⌉=14\lceil 1120/80 \rceil = 14 chips. Real runs cut that bill with sharded or low-precision optimizer states, but the shape of the answer never changes: the model trains on many chips or it does not train.

Code

The point is the check: replay the memory arithmetic instead of quoting it.

params = 70_000_000_000chip_gb = 80
weights_gb = params * 2 / 1e9print(f"weights: {weights_gb:.0f}GB")  # -> weights: 140GB
import mathprint(f"chips for weights: {math.ceil(weights_gb / chip_gb)}")  # -> chips for weights: 2
adam_gb = params * 12 / 1e9total_gb = weights_gb + weights_gb + adam_gb  # weights + gradients + Adam statesprint(f"full training state: {total_gb:.0f}GB")  # -> full training state: 1120GBprint(f"chips for full state: {math.ceil(total_gb / chip_gb)}")  # -> chips for full state: 14
grads_gb = 7_000_000_000 * 2 / 1e9print(f"all-reduce per GPU: {grads_gb * 1.5:.0f}GB")  # -> all-reduce per GPU: 21GB

Drop the parameter count to 7 billion and the weights fit one chip at 14GB, which is exactly when plain data parallelism becomes the whole answer.

Watch Out For

Adding GPUs until the run gets slower

GPUs compute fast but the cables between them move data slowly. Past some count, each step spends more time synchronizing than calculating, and throughput falls as chips are added. The symptom is scaling efficiency sliding under 50 percent while utilization dashboards show GPUs waiting on communication. Fix: measure time per step against GPU count before renting more, and reach for gradient accumulation to grow the effective batch on the chips already paid for.

Splitting layers and forgetting the bubble

Pipeline stages look perfectly balanced on paper while hardware sits idle in practice. The symptom is a utilization trace shaped like a staircase: early GPUs dark at the end of each step, late GPUs dark at the start. Fix: raise the micro-batch count so the pipe stays full, and confirm with per-stage timelines rather than average utilization, which hides the bubble.

The Quick Version

  • A 70-billion parameter model needs 140GB for weights at 16-bit, which is 1.75 times an 80GB chip, so weights alone span at least 2 chips.
  • Full Adam states add about 840GB, taking weights plus gradients plus states to 1120GB, or 14 chips at 80GB each.
  • Data parallelism copies the model to every GPU and splits batches, then synchronizes gradients with an all-reduce.
  • A 4-GPU all-reduce on a 7-billion parameter model moves 21GB per GPU per step, 1.5 times the 14GB gradient size.
  • Pipeline parallelism splits layers across chips and pays a bubble of idle time; tensor parallelism splits matrices and demands the fastest links.
  • Past some GPU count, communication dominates and runs slow down, so scaling efficiency decides the chip count, not ambition.