Skip to content
AI360Xpert
Glossary
Definition

Multi-Token Prediction

A training technique that adds extra heads predicting several future tokens per position, not just the next one, then discards those heads at inference.

Ordinary next-token training only ever rewards a model for guessing one step ahead. Multi-token prediction widens that signal during training by attaching a few extra heads on the same shared representation, each predicting further into the future — t+2t{+}2, t+3t{+}3, and so on — alongside the usual t+1t{+}1 head.

Every head but the ordinary one is discarded once training finishes, so the deployed model generates exactly like a normal autoregressive model. Some systems keep the extra heads anyway, reusing them as a draft model for speculative decoding.