Prefix Caching
A performance optimization technique that stores the computed KV cache of common system prompts to instantly reuse them across multiple inference requests.
Think of It Like This
Like a barista pre-measuring exactly 100 cups of the house blend every morning so they don't have to grind beans for every single customer.
In production, many users interact with an LLM using the exact same massive system prompt (e.g., 'You are a helpful coding assistant...'). By calculating and caching the KV vectors for this prefix once, the inference engine can skip processing it for subsequent requests. This drastically reduces time-to-first-token (TTFT) and saves massive amounts of compute.