Skip to content
AI360Xpert
Core ML

Continuous Batching

Instead of waiting for an entire batch of users to finish their long text generation before starting a new batch, the server ejects finished users instantly and injects new users mid-flight. It's like a revolving door for GPU memory.

In traditional batching, the GPU waits for the longest sentence to finish before taking new requests. In Continuous Batching (Iteration-level scheduling), the moment User A finishes, their slot in the batch is immediately given to User C.
In traditional batching, the GPU waits for the longest sentence to finish before taking new requests. In Continuous Batching (Iteration-level scheduling), the moment User A finishes, their slot in the batch is immediately given to User C.

Why Does This Exist?

In classic dynamic-batching, the server gathers 32 user requests into a single massive matrix, sends it to the GPU, and waits for the GPU to finish processing all 32 requests before it gathers the next batch.

This works perfectly for Image Recognition, because every image takes the exact same amount of time to process.

However, Large Language Models (LLMs) generate text one word (token) at a time.

  • User A asks: "What is 2+2?" (The model answers in 3 tokens: "It is 4.")
  • User B asks: "Write a 1,000 word essay on the French Revolution." (The model answers in 1,200 tokens).

If User A and User B are put in the same standard batch, User A's generation finishes in 0.1 seconds. But User A's slot in the GPU is locked until User B finishes 10 seconds later! For 9.9 seconds, the GPU is wasting massive amounts of compute doing absolutely nothing for User A's slot. This flaw made early LLMs incredibly expensive to serve.

Continuous Batching (also known as Iteration-level scheduling or In-flight batching) solves this by scheduling at the token level, not the request level.

Think of It Like This

Think of It Like This

Imagine a restaurant table with 4 seats (the GPU batch size).

Standard Batching: 4 strangers are seated together. Person A finishes eating in 10 minutes. Person B eats slowly and takes an hour. Person A is not allowed to leave the table, and no new customers can sit down, until Person B finally finishes.

Continuous Batching: The restaurant uses a revolving door policy. The exact second Person A finishes their sandwich, they leave, and a new customer from the waiting line immediately sits in their chair and starts eating, while Person B continues their slow meal.

How It Actually Works

Continuous batching was pioneered by a system called Orca (and popularized by frameworks like vLLM and TGI).

The LLM generation process happens in a giant while loop, predicting one token at a time. Instead of the batch being a static, unchangeable matrix, the Inference Engine re-evaluates the batch every single time the loop ticks (iteration-level).

  1. Iteration 1: The batch contains User A and User B. The GPU predicts 1 token for each.
  2. Iteration 2: User A's sequence hits the <EOS> (End of Sequence) token. They are done.
  3. Iteration 3 (The Swap): The Inference Engine instantly removes User A's data from the GPU's memory. It looks at the incoming HTTP queue, grabs User C, and injects User C's prompt into the empty slot in the matrix.
  4. Iteration 4: The GPU predicts 1 token for User B, and the first token for User C, simultaneously.

Show Me the Code

You cannot implement continuous batching in a standard PyTorch while loop without rewriting the core attention mechanism (usually relying on custom CUDA kernels like PagedAttention).

Instead, you get Continuous Batching for free by running an advanced Inference Engine like vLLM. You do not write the batching logic; you just run the server.

# 1. Install vLLMpip install vllm
# 2. Launch your LLM. # vLLM automatically handles Continuous Batching under the hood.python -m vllm.entrypoints.openai.api_server \    --model meta-llama/Llama-2-7b-chat-hf \    --max-num-batched-tokens 4096

When you send HTTP requests to this server, vLLM will automatically inject new incoming requests into the GPU mid-flight.

Watch Out For

Watch Out For

The KV Cache Memory Explosion. To make Continuous Batching work, the server has to keep track of the history (the KV Cache) of every user currently mid-flight. If the server aggressively injects 200 concurrent users into the batch, the memory required to hold 200 unique KV Caches will easily exceed the physical limits of the GPU's VRAM, causing an Out of Memory (OOM) crash. Continuous batching only works if it is paired with hyper-efficient KV Cache Management (like PagedAttention) that safely pages memory in and out.

The Quick Version

  • Standard Dynamic Batching forces fast requests to wait for slow requests to finish, wasting massive amounts of GPU compute when generating text.
  • Continuous Batching operates at the token level (iteration-level scheduling).
  • The exact millisecond a short request finishes, the server ejects it and injects a new queued request into the empty slot in the batch matrix.
  • It can increase the maximum throughput of an LLM server by 20x to 40x.
  • You do not code this yourself; you use an engine like vLLM, TGI, or TensorRT-LLM.
  • inference-engines — The specialized C++ and Rust servers that actually run this complex logic.
  • kv-cache-management — The memory system (PagedAttention) required to keep Continuous Batching from crashing the GPU.

Related concepts