KV Cache Management
Instead of forcing the LLM to re-read the entire conversation from scratch every time it wants to generate a new word, the server saves the mathematical summary of the conversation in the GPU's memory. PagedAttention manages this memory just like a computer's operating system.
Why Does This Exist?
When a Large Language Model generates text, it predicts one token at a time. If you give it a prompt that is 1,000 tokens long, it generates the 1,001st token. To generate the 1,002nd token, a naive system would feed all 1,001 tokens back into the model from scratch, re-calculating the massive Attention matrices for all 1,001 tokens again.
Re-calculating the history every single step is computationally impossible at scale. Instead, we use a KV (Key-Value) Cache. As the model calculates the internal mathematical representations (Keys and Values) for the first 1,000 tokens, it saves those matrices in the GPU's memory (VRAM). When calculating the 1,002nd token, it only computes the math for the 1,001st token, and simply looks up the rest of the history from the KV Cache.
The problem is that the KV Cache is absolutely massive. Serving a 13B parameter model to just 100 concurrent users can easily require 100GB of VRAM just for the KV Cache.
Think of It Like This
Think of It Like This
Imagine you are reading a 500-page novel.
Without a KV Cache, every time you want to read a new page, you are forced to re-read pages 1 through 499 first to remember the story.
With a KV Cache, you write a short summary of the story in a notebook (VRAM) as you read. When you get to page 500, you just read your notebook summary and then read page 500. The problem is, if 100 people are reading the novel at the same time, you need 100 notebooks, and the library (the GPU) runs out of shelf space.
The Memory Fragmentation Problem
Before 2023, systems managed KV Cache memory poorly. If User A started a prompt, the system had to guess how long their conversation would be. It would pre-allocate a massive, contiguous block of memory (e.g., 2,048 tokens). If User A only generated 50 tokens, the remaining 1,998 tokens of memory were completely wasted. This is called Internal Fragmentation. Up to 60% of GPU memory was wasted because of this pre-allocation!
The Solution: PagedAttention (vLLM)
In 2023, researchers at UC Berkeley released PagedAttention (the core technology behind vLLM), which completely revolutionized LLM serving.
Instead of allocating one giant, contiguous block of memory per user, PagedAttention steals an idea from Operating Systems: Virtual Memory Paging.
- Blocks: The KV Cache is divided into tiny, fixed-size blocks (e.g., each block holds exactly 16 tokens).
- Non-Contiguous Allocation: When User A starts generating text, they are given one block. When that block fills up, they are given a second block. The second block does not need to be physically next to the first block in VRAM.
- The Block Table: A mapping table keeps track of which physical blocks belong to User A.
Because memory is allocated strictly on-demand in tiny chunks, memory waste drops from 60% to under 4%.
The Bonus: Zero-Copy Sharing
Because memory is broken into discrete blocks, multiple users can actually share blocks. If 50 users all use the exact same System Prompt ("You are a helpful assistant..."), the KV Cache for that prompt is only computed and stored once in a physical block. All 50 users' Block Tables simply point to that same shared block, saving enormous amounts of memory. This is critical for complex prompt-chaining and agentic workflows.
Show Me the Code
You do not write the CUDA kernels for PagedAttention yourself. You benefit from it automatically by using an inference engine like vLLM or TensorRT-LLM.
When launching the engine, you configure the block size.
# Launching vLLM with specific PagedAttention configurationspython -m vllm.entrypoints.openai.api_server \ --model mistralai/Mistral-7B-Instruct-v0.1 \ --block-size 16 \ --gpu-memory-utilization 0.90--block-size 16 tells PagedAttention to chunk the KV Cache into blocks of exactly 16 tokens. --gpu-memory-utilization 0.90 tells the engine to dedicate exactly 90% of the GPU's VRAM entirely to the KV Cache holding area.
Watch Out For
Watch Out For
Context Window Inflation.
Many modern models advertise massive context windows (e.g., 128k or 1 Million tokens). Do not assume you can actually serve this in production easily! A single user generating a 128,000-token conversation on a 70B model requires roughly 25GB of KV Cache memory for themselves alone. Even with PagedAttention eliminating waste, the sheer absolute size of a 128k KV Cache means you might only be able to serve 2 or 3 concurrent users on an $30,000 GPU before you hit an Out of Memory (OOM) error. To solve this, you must look into model-quantization for the KV Cache.
The Quick Version
- Generating text requires saving the mathematical history of the conversation in the GPU's VRAM. This is the KV Cache.
- In standard systems, predicting the size of the KV Cache caused massive pre-allocation and memory fragmentation (60% waste).
- PagedAttention (used in
vLLM) divides the KV Cache into tiny, non-contiguous blocks that are allocated on-demand, reducing waste to near zero. - It also allows multiple users to share the exact same memory blocks if they share the same prompt (Zero-Copy Sharing).
- Managing the massive size of the KV Cache is the absolute hardest part of scaling an LLM product.
What to Read Next
continuous-batching— Why PagedAttention is the exact technology that made Continuous Batching computationally viable.model-quantization— How to compress the KV Cache from 16-bit floats down to 8-bit or 4-bit, doubling the number of users you can serve.