LLM Chat Assistant
AdvancedOverview
An LLM Chat Assistant (like ChatGPT or Claude) is a conversational Gen AI application. It provides stateful, natural language interactions by routing user prompts to Large Language Models (LLMs). The central challenges include managing conversation history, filtering content for safety, routing to the optimal model based on workload, and streaming token-by-token responses back to the client at massive scale.
Functional Requirements
- Users can start new chat sessions and send text prompts.
- The system must maintain conversational context (memory) across multiple turns in a session.
- The response must be streamed back to the client token-by-token in real-time.
- Prompts and outputs must be filtered for safety and policy violations.
- Users can view their historical chat sessions.
Non-Functional Requirements
- Time to First Token (TTFT) should be under ~500ms at p99.
- Time Between Tokens (TBT) should be low enough to read fluidly (e.g., < 50ms).
- High availability and fault tolerance for the stateless orchestration layer.
- Efficient GPU utilization, batching concurrent requests to maximize throughput.
Capacity Estimation
Assume 100M Daily Active Users (DAU), each sending 10 prompts/day on average. Each prompt averages 100 tokens, and the generated response averages 400 tokens.
- Traffic: 100M x 10 = 1 Billion prompts/day. 1B / 86,400s ≈ 11,500 requests/sec.
- Compute (Inference): LLM inference is bound by memory bandwidth and compute. At 11.5k RPS, assuming a 70B parameter model, the system requires thousands of GPUs (e.g., H100s or A100s) partitioned across multiple clusters. Continuous batching and paged attention are mandatory to achieve reasonable throughput.
- Storage (History): Each turn (prompt + response) ≈ 1KB. 1B turns/day = 1TB/day. Over a year, this is 365 TB of conversational text data.
High-Level Architecture
The architecture relies on a stateless orchestration layer (Chat Service) communicating with stateful backend stores and compute clusters. When a user sends a prompt, the API Gateway routes it to the Chat Service. The service fetches the recent conversation history from a NoSQL database and passes the full context to a Moderation Service. Once cleared, the payload is sent to the Inference Gateway, which queues and batches requests for the Inference Cluster (GPUs). As the LLM generates tokens, they are streamed back through the Inference Gateway to the Chat Service, which multiplexes the stream: sending tokens back to the user via WebSockets/SSE, and asynchronously persisting the final response to the database.
Data Model
| Entity | Fields / Schema | Storage Choice |
|---|---|---|
| user | user_id (PK), email, created_at, subscription_tier | Relational DB (PostgreSQL) |
| chat_session | session_id (PK), user_id (Index), title, created_at, updated_at | Relational DB (PostgreSQL) or NoSQL |
| message | message_id (PK), session_id (Index), role (user/assistant), content, created_at, token_count | Wide-column store (Cassandra) or Document DB (DynamoDB) |
Detailed Design
Context Window Management
LLMs are inherently stateless and have a fixed context window limit. The Chat Service must fetch historical messages from the database and append the new prompt. If the total token count exceeds the model's limit (or a strict budget), the service must truncate older messages, summarize them, or use a sliding window approach before sending the payload to the model.
Inference Engine (Continuous Batching & PagedAttention)
Serving LLMs efficiently requires specialized inference engines like vLLM or TensorRT-LLM. Because output lengths are unpredictable, static batching wastes GPU cycles. Continuous Batching allows new requests to join a batch as soon as an older request finishes generating. PagedAttention manages the KV cache (key-value tensors for past tokens) efficiently by treating memory like an OS virtual memory page table, eliminating memory fragmentation and allowing sharing of system prompts.
Streaming Responses (SSE / WebSockets)
Given that generating 400 tokens might take 5-10 seconds, waiting for completion before responding provides a terrible UX. The inference engine yields tokens one by one. The Chat Service consumes this stream and pushes it to the client using Server-Sent Events (SSE) or WebSockets, creating the "typing" effect.
Safety and Moderation
Both the input prompt and the output stream must be evaluated. To avoid delaying the Time to First Token (TTFT), input moderation can happen in parallel with the initial inference request. Output moderation often involves a smaller, faster classifier analyzing sliding windows of the output stream. If a violation is detected mid-stream, the connection is abruptly closed with a standard refusal message.
Bottlenecks & Solutions
The primary bottleneck is GPU Memory Bandwidth during the decoding phase. While the prefill phase (processing the input prompt) is compute-bound, generating tokens one-by-one is memory-bandwidth bound because the entire model weights and KV cache must be loaded from HBM to SRAM for every single token. Horizontal scaling of inference nodes and utilizing model parallelism (Tensor Parallelism and Pipeline Parallelism) across multiple GPUs are critical solutions.