Speculative Decoding
An inference acceleration technique where a small, fast draft model predicts several tokens ahead, and a larger target model verifies them in parallel.
Think of It Like This
Like an assistant quickly drafting a whole paragraph, and the boss reading it instantly to approve or rewrite it all at once.
Because LLM inference is severely memory-bandwidth bound rather than compute bound, the large model has spare capacity to verify 5 tokens simultaneously. If the draft model's predictions are correct, the system gets 5 tokens for the time cost of 1. If incorrect, it rejects the draft and corrects it seamlessly without any loss in output quality.