Sequence Parallelism
When your context window gets so large that the activations for a single sequence no longer fit on one GPU, you must split the sequence itself across multiple GPUs.
The Memory Bottleneck of Long Contexts
As models push toward 1-million-token context windows, a new memory bottleneck emerges: Activation Memory.
While tensor-parallelism solves the problem of model weights being too large, it does not solve the activation problem. In standard Tensor Parallelism (TP), every GPU in the TP group must store the entire sequence of activations. If your sequence is 1 million tokens long, the memory required to store the activations (even just the LayerNorm and Dropout outputs) exceeds the capacity of an 80GB GPU.
What is Sequence Parallelism (SP)?
Introduced in Megatron-LM, Sequence Parallelism (SP) is an extension of Tensor Parallelism. Instead of every GPU storing the full sequence, the sequence dimension itself is sharded across the GPUs.
If you have 4 GPUs and a sequence of 100,000 tokens:
- GPU 0 processes tokens 1 to 25,000
- GPU 1 processes tokens 25,001 to 50,000
- GPU 2 processes tokens 50,001 to 75,000
- GPU 3 processes tokens 75,001 to 100,000
How It Works
Sequence Parallelism relies on the fact that certain operations in a Transformer are completely independent across the sequence dimension.
- Independent Operations (LayerNorm, Dropout, GeLU): These operations act on each token individually. They don't need to see the other tokens. In SP, these operations are executed on the sharded sequence locally on each GPU, slashing activation memory by a factor of (where is the number of GPUs).
- Dependent Operations (Self-Attention, Linear Projections): Self-attention requires tokens to look at each other. Linear projections require matrix multiplications that are already sharded by TP.
The All-Gather / Reduce-Scatter Dance
To make SP work seamlessly with TP, the system uses two communication primitives:
- All-Gather: Before the TP matrix multiplications, the GPUs gather the full sequence from the shards so the matrix math can proceed normally.
- Reduce-Scatter: After the TP operations, the results are summed and scattered back into sequence shards.
This completely replaces the standard All-Reduce operations in TP, meaning Sequence Parallelism does not add any additional communication overhead. It simply reshapes the existing communication while saving massive amounts of memory.
The Ring Attention Alternative
For truly massive contexts (e.g., 10M tokens), even gathering the sequence temporarily for attention (via All-Gather) crashes the GPU. In these extreme cases, engineers use ring-attention, which passes blocks of keys and values in a circle around the GPUs, ensuring the full sequence never materializes on a single device.
The Quick Version
- Sequence Parallelism shards the input sequence across GPUs to save activation memory.
- It is essential for training models with massive context windows (100k+ tokens).
- It applies to operations that are independent per-token, like LayerNorm and Dropout.
- It integrates perfectly with Tensor Parallelism by replacing All-Reduce with All-Gather and Reduce-Scatter, adding zero extra communication overhead.