Core ML

LLM Application Architecture

Calling an LLM API directly is not a production system. A real LLM application wraps that call in a stack of layers — gateway, cache, router, guardrails, and observability — each one solving a problem that raw inference can't.

An LLM application is not a single model call. Every request flows through a Gateway, a Cache check, a Router, the Model, then Guardrails, with an Observability bus collecting spans from every layer.

Why Does This Exist?

The moment you move an LLM from a demo to production, a single openai.chat.completions.create() call stops being enough. You immediately hit problems the API doesn't solve: which requests should hit the expensive GPT-4o versus the cheap GPT-4o-mini? What happens when the API returns a 503? How do you catch a response that tells the user how to synthesize something dangerous? How do you know, three days later, which prompt caused the mysterious accuracy drop?

Each of those questions has an answer — but each answer is a component. Put them together and you have an LLM application architecture.

Think of It Like This

Think of an airport. The plane (the model) is what actually moves you. But before you touch the plane, you pass through check-in (the gateway), security screening (guardrails), and a departure board (the router) that tells you which gate to go to. Every step is logged by the airline's system (observability). Remove any layer and something breaks: no security means unsafe flights; no router means passengers boarding the wrong plane.

How It Actually Works

A production LLM request travels through several layers in sequence.

The Gateway is the front door. It enforces authentication (does this API key have permission?), rate limiting (has this tenant sent too many requests this minute?), and request logging. Without a gateway, a single misconfigured client can exhaust your entire model budget overnight.

The Cache intercepts the request before it reaches the model. If the exact same prompt — or a semantically close one — was answered five minutes ago, return that answer instantly for near-zero cost. Three caching strategies exist: exact match (fast, narrow), prefix match (good for system-prompt-heavy apps), and semantic match (fuzzy, using embeddings). See llm-caching for the tradeoffs between them.

The Router looks at what the request actually is and decides which model handles it. A two-sentence factual question goes to a small fast model. A long reasoning task with code review goes to the large expensive one. A multilingual request might go to a specialist model. Routing cuts costs by 40–60% in typical production traffic without any accuracy loss for the easy cases.

The Model is where inference happens. This is the only layer you can't swap out — everything else is wrapping it.

Guardrails inspect the model's output before it reaches the user. They catch hallucinated citations, toxic language, PII leaks, and off-topic responses. Some teams also run guardrails on the input to catch prompt injection before the model sees it.

The Observability Bus is not sequential — it's a sidecar that every layer writes to. Every layer emits a span: how long it took, what it sent, what it received, how many tokens it consumed, and what it cost. Without this, debugging a production issue means guessing.

The Store Layer

Most diagrams omit this, but a real application also has a Store layer behind the model: a short-term memory (conversation history), a long-term vector store (the knowledge base for RAG), and a tool result cache. The model reads from and writes to the store on every multi-turn conversation.

Show Me the Code

A minimal gateway-cache-model pattern in Python using LiteLLM:

1import litellm2from cachetools import TTLCache3
4# Simple exact-match cache with 5 minute TTL5_cache: TTLCache = TTLCache(maxsize=1000, ttl=300)6
7def call_llm(prompt: str, model: str = "gpt-4o-mini") -> str:8    # 1. Cache check9    key = f"{model}:{hash(prompt)}"10    if key in _cache:11        return _cache[key]  # free response in <1ms12
13    # 2. Route: long prompts get the big model14    if len(prompt.split()) > 200:15        model = "gpt-4o"16
17    # 3. Model call with automatic fallback to anthropic18    response = litellm.completion(19        model=model,20        messages=[{"role": "user", "content": prompt}],21        fallbacks=["claude-3-5-sonnet-20241022"],  # retries on 5xx22        num_retries=2,23    )24    result = response.choices[0].message.content25
26    # 4. Store in cache27    _cache[key] = result28    return result

Watch Out For

Observability gaps between layers. The most common production failure mode isn't the model — it's a silent failure in an outer layer. A guardrail that rejects 20% of responses, a cache that's silently corrupted, a router that's sending everything to the slow model because a config flag flipped. If each layer doesn't emit structured logs with a shared request_id, you'll never find these. Wire up the observability bus before you worry about optimizing the model.

The Quick Version

A production LLM app has six layers: Gateway, Cache, Router, Model, Guardrails, and an Observability Bus.
The Gateway handles auth and rate limiting. The Cache saves money on repeated prompts. The Router picks the cheapest model that can handle the request.
Guardrails run on the output before users see it — and optionally on the input to block injection.
Every layer writes to a shared observability bus so you can reconstruct exactly what happened on any given request.