Skip to content
AI360XpertBeta

Tensor Parallelism

When one layer is too big for a single card, split the matrix itself. Each card multiplies its own half, and one collective adds the partial answers.

One matrix multiplication is split across two cards that each compute a partial product, then a single collective adds the halves into the exact full result
One matrix multiplication is split across two cards that each compute a partial product, then a single collective adds the halves into the exact full result

Why Does This Exist?

Copying the whole model to every card stops working the moment the model stops fitting on one card. A 70 billion parameter model needs about 140 gigabytes just to hold its weights in 16 bit precision, and a flagship 80 gigabyte card cannot load it at all. No batch trick gets around that. The weights themselves have to live somewhere.

So the model gets cut up. Data parallelism copies the model and splits the data, which is the easy case and the one to prefer whenever it fits. Tensor parallelism splits the math instead: a single layer, a single matrix, spread across cards that compute side by side and reconcile at the end of the layer. This page covers that intra-layer split. The broader map of strategies sharing the work lives in distributed training.

Think of It Like This

Two clerks splitting one giant invoice

One invoice is too wide for either desk, so two clerks split the columns. The left clerk totals the left columns, the right clerk totals the right columns, then they call out their subtotals across the room and add them. Neither ever held the whole sheet, yet the final total matches to the cent.

The calling out is the point the analogy must carry: it works because they sit in the same room. Put the clerks in different cities and the phone calls cost more than the splitting saves.

Where the picture stops: clerks can split any columns, while cards must split where the math allows it. Cut in the wrong place and the halves need to talk constantly instead of once, which is why the column-then-row pattern below exists.

How It Actually Works

Column split, then row split

Take the feed-forward pair Y=XAY = XA followed by Z=YBZ = YB. Here XX is the layer input with one row per token, AA is the first weight matrix, BB is the second, and the activation between them runs on each shard locally. The standard pattern, popularized as the Megatron-LM split, cuts AA by columns and BB by rows.

Give card one the left half AleftA_{left} and card two the right half ArightA_{right}. Both cards hold a full copy of the small input XX. Card one computes Yleft=XAleftY_{left} = X A_{left}, card two computes Yright=XArightY_{right} = X A_{right}, and neither needs the network yet. Then card one multiplies its own partial result by its own top half BtopB_{top} to get Z1Z_1, card two does the same with BbottomB_{bottom} to get Z2Z_2. The true output is the sum Z1+Z2Z_1 + Z_2, so the cards run one all-reduce, a collective that adds every card piece and hands the total back to all of them. That single sync per layer is the whole communication bill for the forward pass, mirrored once more on the way back.

Worked split with small numbers

Input X=[2,3]X = [2, 3], weights A=[[1,0,4,1],[0,2,1,3]]A = [[1, 0, 4, 1], [0, 2, 1, 3]]. The full product is [2,6,11,11][2, 6, 11, 11]: first entry 2×1+3×0=22 \times 1 + 3 \times 0 = 2, second 2×0+3×2=62 \times 0 + 3 \times 2 = 6, third 2×4+3×1=112 \times 4 + 3 \times 1 = 11, fourth 2×1+3×3=112 \times 1 + 3 \times 3 = 11.

Split AA into left [[1,0],[0,2]][[1, 0], [0, 2]] and right [[4,1],[1,3]][[4, 1], [1, 3]]. Card one computes [2,6][2, 6], card two computes [11,11][11, 11]. Second weights B=[[1,2],[0,1],[1,0],[2,1]]B = [[1, 2], [0, 1], [1, 0], [2, 1]] split into top [[1,2],[0,1]][[1, 2], [0, 1]] and bottom [[1,0],[2,1]][[1, 0], [2, 1]]. Card one computes [2×1+6×0,2×2+6×1]=[2,10][2 \times 1 + 6 \times 0, 2 \times 2 + 6 \times 1] = [2, 10]. Card two computes [11×1+11×2,11×0+11×1]=[33,11][11 \times 1 + 11 \times 2, 11 \times 0 + 11 \times 1] = [33, 11]. The all-reduce adds them: [2+33,10+11]=[35,21][2 + 33, 10 + 11] = [35, 21], exactly the full YBY B. Sixteen weights total, eight per card, one addition to reconcile.

The one collective and why it stays local

That sync fires every layer, forward and backward, so the fabric between the cards sets the speed limit. Inside one server the cards share fast copper links and the addition is quick. Across servers over ordinary networking the cards spend their time waiting on each other instead of multiplying. So tensor parallelism stays within a single fast-interconnect box, with a degree no larger than the cards in that box, while pipeline splits and data copies cover the rest of the cluster. Memory tricks like activation checkpointing shrink what each card must store, which keeps the split degree as small as it can be.

Watch Out For

More split cards than the layer needs

Symptom: you widen the split to fill the box and training gets slower, not faster. Utilization drops while collective time climbs, because each added card contributes another voice to every per-layer sum.

Fix: keep the degree at the smallest value that fits the layer, and spend remaining cards on pipeline stages or data copies instead. If doubling the degree barely trims per-card memory but clearly grows sync time, you have already passed the useful point.

A split stretched across a slow network

Symptom: cards sit idle in lockstep, step time tracks the network rather than the math, and adding bandwidth helps more than adding cards. Profiling shows collectives dominating every layer.

Fix: confine the split to cards joined by the fast in-box fabric, and cross box boundaries with pipeline stages, which talk far less often. Measure the share of step time spent in collectives before and after any placement change.

The Quick Version

  • Tensor parallelism splits one matrix across cards when the layer itself does not fit on a single card.
  • The first matrix splits by columns and the second by rows, so each card computes a partial product independently.
  • One all-reduce per layer adds the partials into the exact full result.
  • The sync fires every layer in both directions, so the split stays inside one fast-interconnect server.
  • Keep the degree minimal and cover the rest of the cluster with pipeline and data splits.