Multi-Tenant Inference
Instead of renting 5 separate GPUs to run 5 separate ML models for 5 separate customers, you load all 5 models onto a single GPU and share the hardware. It's like an apartment building for neural networks.
Why Does This Exist?
You are building an AI startup that offers custom, fine-tuned LLMs for enterprise customers. Customer A (a law firm) gets a legal-tuned model. Customer B (a hospital) gets a medical-tuned model.
If you use a Single-Tenant architecture, you rent one GPU server for Customer A, and a second GPU server for Customer B. If you have 100 customers, you must rent 100 GPUs (costing \72,000$ a day!). Furthermore, if Customer A only sends 10 requests a day, their dedicated GPU sits idle 99% of the time. This is financially unsustainable.
To build a profitable ML business, you must use a Multi-Tenant architecture. You rent a single massive GPU (like an H100) and load Customer A's model, Customer B's model, and Customer C's model into the same GPU memory. When a request comes in, the server routes it to the correct model. You just reduced your hosting costs by 100x.
Think of It Like This
Think of It Like This
Imagine you run a private chef business.
Single-Tenant: You build a completely separate kitchen, buy a separate stove, and hire a separate chef for every single family you cook for.
Multi-Tenant: You build one massive commercial kitchen. You have one chef and one giant stove, but you cook the Smith family's dinner in Pot A, and the Johnson family's dinner in Pot B. You share the underlying infrastructure to maximize efficiency.
How It Works: Multi-LoRA Serving
Loading 100 entirely different 70B models into a single GPU is impossible because you will instantly run out of VRAM. Multi-tenant architecture relies heavily on Parameter-Efficient Fine-Tuning (PEFT), specifically LoRA (Low-Rank Adaptation).
When you fine-tune a model using LoRA, you do not change the massive 140GB Base Model. Instead, you train a tiny 100MB "Adapter" file that contains just the new knowledge.
In a multi-tenant inference engine (like vLLM or LoRAX):
- The Base Model: The massive 140GB base model is loaded into VRAM exactly once.
- The Adapters: Customer A's 100MB adapter and Customer B's 100MB adapter are also loaded into VRAM.
- The Execution: When a request arrives for Customer A, the engine passes the data through the Base Model, and then routes it through Customer A's specific adapter.
Because the adapters are so small, you can literally load thousands of different customer models onto a single GPU.
Dynamic Adapter Swapping
What if you have 10,000 customers? Even 100MB adapters will eventually fill up the VRAM. Modern multi-tenant engines use Dynamic Adapter Swapping.
They keep the Base Model in the GPU VRAM permanently. But they keep the 10,000 adapters in the cheap, slow CPU RAM. When Customer #8,492 sends a request, the engine instantly copies their specific 100MB adapter from the CPU RAM into the GPU VRAM (which takes just a few milliseconds), runs the inference, and then deletes it from the GPU.
Show Me the Code
You can serve multiple LoRA adapters simultaneously using vLLM.
# 1. Start the server with the Base Model# Enable the LoRA feature and allocate memory for adapterspython -m vllm.entrypoints.openai.api_server \ --model meta-llama/Llama-2-7b-hf \ --enable-lora \ --max-loras 4 # Keep up to 4 adapters active in VRAM at once
# 2. Query the server, specifying WHICH adapter you want to use!curl http://localhost:8000/v1/completions \ -H "Content-Type: application/json" \ -d '{ "model": "customer_a_legal_adapter", "prompt": "The defendant is guilty of...", "max_tokens": 50 }'Watch Out For
Watch Out For
The Noisy Neighbor Problem. In a multi-tenant system, all customers share the exact same GPU compute cores. If Customer A suddenly sends a massive spike of 10,000 requests, they will monopolize the GPU's time. Customer B, who only sent 1 request, will suddenly experience terrible latency and timeouts. To prevent this, your REST API gateway must implement strict Rate Limiting and fair-share queueing before the requests ever reach the Inference Engine.
The Quick Version
- Single-Tenant inference gives every customer a dedicated GPU, which is incredibly expensive and leads to idle hardware.
- Multi-Tenant inference loads multiple models onto a single GPU to maximize utilization and reduce costs.
- This is achieved using LoRA Adapters. The massive Base Model is loaded once, and tiny customer-specific adapters are loaded alongside it.
- Dynamic Swapping allows the server to pull adapters from CPU RAM into GPU VRAM on-demand in milliseconds.
- You must implement API Rate Limiting to prevent one customer from slowing down the GPU for everyone else.
What to Read Next
autoscaling-inference— How to scale up your multi-tenant cluster when the shared GPU finally hits 100% capacity.rest-api-serving— Where you actually implement the Rate Limiting logic to prevent the Noisy Neighbor problem.