Skip to content
AI360XpertBeta

Pipeline Parallelism

Split a network by depth across GPUs like an assembly line. Each device runs its own layers and hands activations to the next, and micro-batches keep every station busy.

Four stages process 8 micro-batches in 11 steps of 10 ms for 110 ms total, where the 3 extra steps are the 30 ms bubble that micro-batching shrinks
Four stages process 8 micro-batches in 11 steps of 10 ms for 110 ms total, where the 3 extra steps are the 30 ms bubble that micro-batching shrinks

Why Does This Exist?

A model too large for one GPU must be split, and the slicing direction decides the network bill. Slicing inside layers, as tensor parallelism does, synchronizes several times per layer and needs very fast links, so it stays inside one server. A thousand GPUs across racks cannot talk that fast.

Two ideas make the rest click. Data parallelism copies the whole model per GPU, which fails exactly when the model does not fit. A stage is a consecutive group of layers living on one device. For the full map of options, see data vs tensor vs pipeline parallelism.

This page covers the depth split: consecutive layers per device, one handoff per boundary, and micro-batches against idleness. The sibling that shards weights instead of layers lives in distributed training. The trick that trades compute for memory inside a stage lives in activation checkpointing.

Think of It Like This

A car assembly line on a slow road

Four stations build cars in order: frame, engine, doors, paint. Stations sit in different towns joined by an ordinary road, so instead of shouting constantly they hand over one finished partial car per step. That single handoff is the whole communication pattern.

One car at a time leaves three stations idle, so the line feeds eight cars bumper to bumper. While paint works on car one, doors works on car two, engine on car three, and frame on car four. Stations are pipeline stages, cars are micro-batches, and the startup plus wind-down where some stations still wait is the bubble.

Where it stops: cars never flow backward, but training sends gradients back through the same stations in reverse, so the line must drain forward and then backward every step.

How It Actually Works

Number the stages 11 to SS, meaning consecutive layer groups on consecutive devices. The forward pass sends activations forward: stage 11 runs layers 11 to 1010, passes one tensor to stage 22, which runs layers 1111 to 2020, and so on. The backward pass sends gradients in reverse along the same path. Communication happens only at stage boundaries, once per handoff, which is why ordinary cross-server links suffice.

The bubble, counted

Naive scheduling pushes one batch through and parks every other stage: with SS stages only one works at a time. Micro-batching splits a batch of 3232 into M=8M = 8 micro-batches of 44 and feeds them back to back. If each stage needs t=10t = 10 ms per micro-batch, the pipeline needs:

steps=M+S−1=8+4−1=11\mathrm{steps} = M + S - 1 = 8 + 4 - 1 = 11 total=11×10=110 msbusy=8×10=80 ms\mathrm{total} = 11 \times 10 = 110\ \mathrm{ms} \quad \mathrm{busy} = 8 \times 10 = 80\ \mathrm{ms}

The 33 extra steps are the bubble: 3030 ms of startup plus drain, about 0.270.27 of the schedule. More micro-batches shrink that fraction further but never to zero, and tiny micro-batches waste device efficiency, so MM is tuned per cluster.

Why depth beats width across racks

A boundary handoff carries one activation tensor per micro-batch. A within-layer split synchronizes partial sums several times per layer instead. That order-of-magnitude gap in message count is the entire reason pipeline stages may sit on different racks while tensor shards may not.

Code

The point is the check: rebuild the 110 ms schedule and the 0.27 bubble from stage and micro-batch counts.

stages: int = 4micros: int = 8step_ms: int = 10
steps: int = micros + stages - 1total_ms: int = steps * step_msbubble_ms: int = (steps - micros) * step_ms
print(f"{steps} {total_ms} {bubble_ms}")  # -> 11 110 30print(f"{bubble_ms / total_ms:.2f}")  # -> 0.27

Raise micros to 16 and rerun: steps become 19, total 190 ms, bubble still 30 ms, fraction 0.16. That falling fraction is the GPipe trick, and the fixed 30 ms floor is why the bubble never vanishes.

Watch Out For

Even layer counts that unbalance memory

Early layers often hold the large embedding table while late layers hold little, so cutting 80 layers into four groups of 20 overloads the first device and starves the last. The job fails on device one while device four sits half empty.

Symptom: out-of-memory on the first stage at batch sizes the other stages handle easily. Fix: profile per-layer memory and cut unevenly, such as 15, 22, 22, and 21 layers, until peak memory matches across stages.

Too few micro-batches for the stage count

Four stages fed two micro-batches spend most of the step starting up and draining, and the measured speedup over one device is barely above one. The schedule, not the hardware, is the bottleneck.

Symptom: adding stages adds almost no throughput while device idle time stays high. Fix: raise the micro-batch count until the bubble fraction (S−1)/(M+S−1)(S - 1) / (M + S - 1) drops under budget, then check that tiny micro-batches have not hurt per-device efficiency.

The Quick Version

  • Pipeline parallelism splits a model by depth, placing consecutive layer groups on consecutive devices.
  • Devices exchange one activation tensor per boundary, so slow cross-server links suffice.
  • One batch at a time leaves all but one stage idle, and that idle span is the pipeline bubble.
  • Our 4 stages with 8 micro-batches of 4 finish in 11 steps of 10 ms, 110 ms total with a 30 ms bubble.
  • More micro-batches shrink the bubble fraction but never remove the startup plus drain floor.
  • Uneven per-layer memory needs uneven cuts, or the first stage runs out of memory first.