Sequence Packing

Padding short samples to the longest one wastes most of your compute on nothing. Packing concatenates them instead, and the attention mask is the catch.

Four padded rows spend most of their 512 slots on nothing while two packed rows holding the same tokens run 88.9 percent full, with knockout ticks marking where each document ends.

Why Does This Exist?

You are fine-tuning on 200,000 support tickets. The context length is 512 tokens and the average ticket runs 114.

Every batch is padded to 512, so roughly three quarters of the token slots your accelerator processes hold a pad token: no information, no gradient, full cost. You pay for four times the compute you use, and the loss curve looks completely normal while you do it.

Packing concatenates samples end to end and cuts the stream into full rows, so nearly every slot holds a real token. It is one of the largest free speedups in language-model training, and it ships with exactly one correctness condition that everyone gets wrong once: two documents now share a row, and attention is all-to-all within a row, so token 300 in the middle of ticket two can read token 50 in the middle of ticket one unless something stops it.

This page covers that trick and that condition. Filtering or truncating over-long inputs is a preprocessing decision, covered with the rest of text preparation in text data preprocessing.

The one idea you need first: the context window is the fixed number of token slots the model reads at once, here 512. A batch is a rectangle, so every row in it must be the same length. Those two facts together are the entire source of the waste: short samples get stretched with padding to fill a rectangle sized for the longest one.

Think of It Like This

Sixty notes in envelopes built for a manuscript

You have sixty short notes to post and a box of envelopes sized for a manuscript. One note per envelope means paying manuscript postage sixty times to move sixty postcards of paper.

So you put ten notes in each envelope. Six envelopes instead of sixty, and almost all the paper you pay to move is writing.

Now the part that matters. Whoever opens the envelope must know where each note ends. Miss that and they read ten notes as one rambling document: every sentence individually correct, the whole thing meaningless, and the last line of note three answering a question from note four. That divider is your attention mask. Packing without one is ten notes loose in an envelope.

How It Actually Works

Where the waste comes from

Batches are rectangular. Every row must match the longest member, so shorter rows carry pad tokens through the full forward and backward pass.

Two facts make it worse than the average suggests. Padding is set by the longest sample in the batch, not the mean, so one 500-token outlier pads the other 31 rows. And attention cost grows quadratically with row length, so a padded row is disproportionately expensive rather than merely wasteful.

The mask is the whole correctness story

After packing, a row holds several documents. Attention must stay inside each document, which means the attention mask becomes block diagonal: one block per document, zeros on every between-block cell. Write $a_{ij}$ for the attention weight from token $i$ to token $j$ , with $d(i)$ the document token $i$ belongs to. The rule is $a_{ij} = 0$ whenever $d(i) \neq d(j)$ : tokens may read within their document and never across the edge.

Get it wrong and nothing errors. The model trains, the loss falls, and it has learned that documents continue into unrelated documents. The damage is subtle and real: at generation time a mild tendency to drift into a new topic mid-answer, because that is what training data did. Modern kernels take a cumulative-offsets argument in the cu_seqlens style, the boundary positions of each document in the row, and handle the blocks natively. Prefer that path over materialising a mask by hand.

Two related details. Position ids must restart at each document, or document two believes it starts at position 300, a situation that never occurs at inference. And the loss mask is a separate tensor from the attention mask: pads leave both, while prompt tokens in instruction tuning leave the loss but stay in attention, because the answer must read the question.

Splitting versus dropping

Some documents exceed the context. Two options: truncate and lose the tail, or split across rows, preserving the text but severing the dependency at the cut. For pretraining, splitting is standard and the lost boundary context is accepted. For instruction data a split example is usually incoherent, so filtering by length beats splitting.

The safer alternative

Length bucketing sorts samples by length and batches similar lengths together, so padding inside a batch stays small. Simpler than packing, no mask changes, and it shines when lengths are bimodal rather than long-tailed. The price is that batches stop being randomly composed, which correlates the gradient inside a batch, so shuffle within buckets and keep buckets wide. Packing wins on throughput. Bucketing wins on safety. On a first run, bucket; on a long pretraining run, packing pays for the care it needs.

Worked example

Eight samples in a batch, token lengths 120, 90, 300, 40, 55, 30, 210, and 65. That is $910$ real tokens against a context of $512$ .

Padded, one sample per row: $8 \times 512 = 4{,}096$ slots processed. Utilisation is $910/4096 = 0.222$ , so over three quarters of the compute goes to pad tokens.

Packed: concatenate to 910 tokens and cut into full rows, giving $\lceil 910/512 \rceil = 2$ rows and $1{,}024$ slots. Utilisation is $910/1024 = 0.889$ .

Four times fewer slots for the same eight samples. The cumulative boundary offsets, cumsum of the lengths cut at each 512 edge, are exactly what the mask needs, which is the other half of doing this correctly.

Code

1import numpy as np2
3lengths: np.ndarray = np.array([120, 90, 300, 40, 55, 30, 210, 65])4context: int = 5125
6def padded_slots(lens: np.ndarray, ctx: int) -> int:7    """One row per sample, every row stretched to the context length."""8    return int(len(lens) * ctx)9
10def packed_slots(lens: np.ndarray, ctx: int) -> int:11    """Concatenate, then cut into full rows."""12    return int(np.ceil(lens.sum() / ctx) * ctx)13
14print(int(lengths.sum()), padded_slots(lengths, context))  # -> 910 409615print(round(float(lengths.sum() / padded_slots(lengths, context)), 3))  # -> 0.22216print(packed_slots(lengths, context))                      # -> 102417print(round(float(lengths.sum() / packed_slots(lengths, context)), 3))  # -> 0.88918print(padded_slots(lengths, context) // packed_slots(lengths, context))  # -> 4

np.cumsum(lengths) gives the boundary offsets the mask consumes. Utilisation jumps from 0.222 to 0.889, and the slot count drops by a factor of 4.

Watch Out For

Packing without a document-boundary mask

You implement packing, throughput jumps four times, and the loss looks better than before because each step now sees four times the real tokens.

Nothing says the mask is wrong. Attention across a boundary is a valid computation on valid tensors, and the model learns from it: every packed row teaches that one document flows into the next. What ships is a model with a slight, persistent tendency to change subject mid-response, and no test names the cause.

Position ids are the twin bug. Leave them running across the row and document two starts at position 300, so the model meets a document that begins mid-window, which never happens at inference. Verify instead of trusting: build one packed row from two documents, run it, run each document alone, and compare the logits on the second document tokens. A correct mask gives identical numbers, and the check takes five minutes.

Confusing the attention mask with the loss mask

Different masks, different jobs, and the names in most codebases do not help.

The attention mask controls what a token may read. The loss mask controls which positions train the model. Pads leave both. Prompt tokens in instruction tuning leave the loss but stay in attention.

Two failures follow. Include pad positions in the loss and the model spends capacity learning to predict padding, diluting every real gradient. Exclude prompt tokens from attention rather than from the loss and the model must produce answers without reading the questions. Keep them as separately named tensors and assert their sums: loss positions must equal your real answer-token count, a one-line check that catches both.

The Quick Version

Batches are rectangular, so every row is padded to the longest member and short samples drag pad tokens through the full pass.
Eight samples totalling 910 tokens at a context of 512: padded is 4,096 slots at 22.2 percent, packed is 1,024 at 88.9 percent. Four times less compute.
Padding follows the longest sample, not the mean, and attention cost is quadratic in row length, so the waste compounds.
Packing puts several documents in a row, so the attention mask must be block diagonal with zeros across every document edge.
A wrong mask never errors. The model learns that documents continue into unrelated documents and drifts mid-answer.
Position ids restart at every document edge. Prefer a kernel that takes cumulative sequence lengths over a hand-built mask.
The attention mask and the loss mask are different tensors. Pads leave both; prompts leave only the loss.
Length bucketing is the safer alternative: less throughput, no mask changes, a sensible default for a first run.
Verify by running one document alone and inside a packed row, then comparing logits.