Model Parallelism

Model parallelism slices the network itself across GPUs, so a vision model too large for one card can still train.

Eight layers split four and four across two GPUs with activations crossing once forward and gradients once back

Why Does This Exist?

Data parallelism copies the whole model everywhere, which fails the moment weights plus optimizer state exceed one GPU. Vision transformers keep crossing that line: bigger backbones, larger feature maps, longer video clips. Buying a bigger card stops working.

Model parallelism slices the model instead of the data. Layers or tensors spread across devices while the same batch flows through them in turn. Data parallelism is the sibling to compare against, large-scale pretraining sets the budget context, and batching decides the micro-batch sizes that keep the pipe full.

Think of It Like This

An assembly line versus eight home workshops

Eight workshops each building whole cars need full toolkits and stall on big trucks. An assembly line splits the build into stations: each station owns fewer tools and every car passes through all of them.

Layers are stations and activations are the chassis moving down the line. The analogy stops at reversibility: cars never run backwards, while training sends gradients back up the same line.

How It Actually Works

Two flavours combine. Pipeline parallelism places whole layers on different GPUs: GPU 0 holds layers 1 to 4, GPU 1 holds 5 to 8, activations cross once forward and gradients once back. Tensor parallelism splits individual matmuls inside a layer across GPUs and all-reduces partial results. Micro-batching keeps the pipe busy by feeding small chunks in turn; without it, GPU 1 idles while GPU 0 works, the pipeline bubble.

Worked example

Eight layers, $2$ GPUs, $4$ layers each. One micro-batch forward takes $4$ units per GPU in sequence, $8$ total, then backward mirrors it. With $4$ micro-batches pipelined, steady state keeps both GPUs busy except a fill plus drain bubble over a $40$ -unit schedule, one fifth idle. Activation memory per GPU halves versus one card, which is exactly what lets the model fit at all.

Code

Shard boundaries and bubble share for a layer split.

1def pipeline_split(n_layers: int = 8, n_gpus: int = 2) -> tuple[list[tuple[int, int]], float]:2    """Layer ranges per GPU and bubble share with 4 micro-batches."""3    per = n_layers // n_gpus  # even split assumption4    ranges = [(i * per + 1, (i + 1) * per) for i in range(n_gpus)]5    bubble = 2.0 * (n_gpus - 1) / (2.0 * (n_gpus - 1) + 4 * 2)  # fill+drain over schedule6    return ranges, bubble7
8
9print(pipeline_split())  # -> ([(1, 4), (5, 8)], 0.2)

Watch Out For

Balancing by layer count instead of cost

Early backbone layers on big feature maps cost far more than late layers on small ones. Equal layer counts then overload GPU 0 while GPU 1 waits, and the bubble math lies.

Profile per-layer time and memory first. Split by cost, not by count.

The Quick Version

Model parallelism slices layers or tensors across GPUs when one card cannot hold the model.
Pipeline splits place layer ranges per device; tensor splits divide matmuls inside layers.
Worked numbers: $8$ layers over $2$ GPUs, $4$ each, one fifth idle (0.2) with $4$ micro-batches.
Micro-batching fills the pipe; without it the bubble dominates.
Balance by measured cost per layer, never by raw layer count.