Tensor Parallelism
When one layer is too big for a single card, split the matrix itself. Each card multiplies its own half, and one collective adds the partial answers.
Why Does This Exist?
Copying the whole model to every card stops working the moment the model stops fitting on one card. A 70 billion parameter model needs about 140 gigabytes just to hold its weights in 16 bit precision, and a flagship 80 gigabyte card cannot load it at all. No batch trick gets around that. The weights themselves have to live somewhere.
So the model gets cut up. Data parallelism copies the model and splits the data, which is the easy case and the one to prefer whenever it fits. Tensor parallelism splits the math instead: a single layer, a single matrix, spread across cards that compute side by side and reconcile at the end of the layer. This page covers that intra-layer split. The broader map of strategies sharing the work lives in distributed training.
Think of It Like This
Two clerks splitting one giant invoice
One invoice is too wide for either desk, so two clerks split the columns. The left clerk totals the left columns, the right clerk totals the right columns, then they call out their subtotals across the room and add them. Neither ever held the whole sheet, yet the final total matches to the cent.
The calling out is the point the analogy must carry: it works because they sit in the same room. Put the clerks in different cities and the phone calls cost more than the splitting saves.
Where the picture stops: clerks can split any columns, while cards must split where the math allows it. Cut in the wrong place and the halves need to talk constantly instead of once, which is why the column-then-row pattern below exists.
How It Actually Works
Column split, then row split
Take the feed-forward pair followed by . Here is the layer input with one row per token, is the first weight matrix, is the second, and the activation between them runs on each shard locally. The standard pattern, popularized as the Megatron-LM split, cuts by columns and by rows.
Give card one the left half and card two the right half . Both cards hold a full copy of the small input . Card one computes , card two computes , and neither needs the network yet. Then card one multiplies its own partial result by its own top half to get , card two does the same with to get . The true output is the sum , so the cards run one all-reduce, a collective that adds every card piece and hands the total back to all of them. That single sync per layer is the whole communication bill for the forward pass, mirrored once more on the way back.
Worked split with small numbers
Input , weights . The full product is : first entry , second , third , fourth .
Split into left and right . Card one computes , card two computes . Second weights split into top and bottom . Card one computes . Card two computes . The all-reduce adds them: , exactly the full . Sixteen weights total, eight per card, one addition to reconcile.
The one collective and why it stays local
That sync fires every layer, forward and backward, so the fabric between the cards sets the speed limit. Inside one server the cards share fast copper links and the addition is quick. Across servers over ordinary networking the cards spend their time waiting on each other instead of multiplying. So tensor parallelism stays within a single fast-interconnect box, with a degree no larger than the cards in that box, while pipeline splits and data copies cover the rest of the cluster. Memory tricks like activation checkpointing shrink what each card must store, which keeps the split degree as small as it can be.
Watch Out For
More split cards than the layer needs
Symptom: you widen the split to fill the box and training gets slower, not faster. Utilization drops while collective time climbs, because each added card contributes another voice to every per-layer sum.
Fix: keep the degree at the smallest value that fits the layer, and spend remaining cards on pipeline stages or data copies instead. If doubling the degree barely trims per-card memory but clearly grows sync time, you have already passed the useful point.
A split stretched across a slow network
Symptom: cards sit idle in lockstep, step time tracks the network rather than the math, and adding bandwidth helps more than adding cards. Profiling shows collectives dominating every layer.
Fix: confine the split to cards joined by the fast in-box fabric, and cross box boundaries with pipeline stages, which talk far less often. Measure the share of step time spent in collectives before and after any placement change.
The Quick Version
- Tensor parallelism splits one matrix across cards when the layer itself does not fit on a single card.
- The first matrix splits by columns and the second by rows, so each card computes a partial product independently.
- One all-reduce per layer adds the partials into the exact full result.
- The sync fires every layer in both directions, so the split stays inside one fast-interconnect server.
- Keep the degree minimal and cover the rest of the cluster with pipeline and data splits.