Sequence Packing
A crucial data optimization technique during LLM training where multiple short documents are concatenated into a single max-length context window.
Think of It Like This
Like tightly packing multiple small packages into a single large shipping container so the cargo ship doesn't waste any empty space.
Without packing, a GPU processing a sequence length of 4096 would waste massive amounts of compute simply padding a 500-token document with zeros. Packing maximizes hardware utilization. A special attention mask is applied to prevent tokens from one document from incorrectly paying attention to tokens in an unrelated packed document.