Skip to content
AI360XpertBeta

LLM Application Architecture

Calling an LLM API directly is not a production system. A real LLM application wraps that call in a stack of layers — gateway, cache, router, guardrails, and observability — each one solving a problem that raw inference can't.

10,000 daily requests split into 3,000 cache hits, 4,900 small model calls and 2,100 large model calls across gateway, cache, router, model and guardrails
10,000 daily requests split into 3,000 cache hits, 4,900 small model calls and 2,100 large model calls across gateway, cache, router, model and guardrails

Why Does This Exist?

A demo calls the model once and shows the answer. Production cannot. The same prompt sent twice bills twice. A long reasoning task on a small model fails quietly. A provider 503 at 2am pages no one. An unsafe answer reaches a user with no record of which prompt caused it.

Each failure needs its own layer. Authentication and rate limits at the front door. A cache for repeats. A router that picks the cheapest model able to do the job. Guardrails that block unsafe output. An observability bus that records what every layer did. This page covers that stack and what each layer buys. The surrounding pipeline lives in ml pipeline architecture. Keeping it healthy lives in model monitoring. Proving it before launch lives in testing ML systems.

Think of It Like This

An airport around one plane

The plane is the model. It does the flying and nothing else. Around it sits an airport. Check in verifies tickets, which is the gateway. A board rebooks frequent flyers without a new flight, which is the cache. A departure board sends short hops to small gates and long hauls to large ones, which is the router. Security stops banned items, which are guardrails. The tower log records every takeoff, which is observability.

Mapping holds part for part. Passengers are requests, gates are model choices, security holds are blocked responses, and the tower log is the shared request id across layers.

Where it stops: airports move people in sequence through people. Here the observability bus runs beside the request, not after it, and cache plus router skip work rather than directing queues. No passenger boards without flying, but most production requests answer without reaching a large model.

How It Actually Works

A request flows in order through five layers while a sixth records everything.

The gateway is the front door. It checks keys, enforces per tenant rate limits, validates shape, and logs the raw request. Without it one misconfigured client can spend a monthly budget overnight.

The cache checks before any inference. Exact match is cheapest. Prefix match reuses a shared system prompt. Semantic match reuses near duplicates by embedding similarity at higher risk of a wrong reuse. A hit answers in milliseconds for near zero cost.

The router picks the model. Short factual asks go small and fast. Long reasoning with code goes large. Multilingual or coded tasks go to a specialist when one exists. Routing is where most production savings come from.

The model runs inference. Deployment choice is local versus hosted, compared in local LLMs versus hosted APIs. Everything else on this page wraps this layer.

Guardrails check output before the user sees it. They block toxic text, leaked personal data, hallucinated citations, and off topic answers. Many teams also screen input for injection before the model sees it.

The observability bus is a sidecar. Every layer emits a span with request id, latency, tokens, cost, and decision. Without a shared id, a flipped router flag or a silently corrupted cache is invisible.

Worked example with 10,000 requests

Use cost units, not vendor prices. Large costs 1010 units per thousand tokens. Small costs 11 unit. Each request averages one thousand tokens.

No stack: 10,000×10=100,00010,000 \times 10 = 100,000 units per day.

With stack: cache hits 3030 percent, so 10,000×0.30=3,00010,000 \times 0.30 = 3,000 answer free. Remaining 7,0007,000 reach the router. Router sends 7070 percent small and 3030 percent large, so 7,000×0.70=4,9007,000 \times 0.70 = 4,900 small and 7,000×0.30=2,1007,000 \times 0.30 = 2,100 large. Cost is 4,900×1+2,100×10=4,900+21,000=25,9004,900 \times 1 + 2,100 \times 10 = 4,900 + 21,000 = 25,900 units. Same traffic falls from 100,000100,000 to 25,90025,900, about a 7474 percent cut, and 3,0003,000 answers skip inference entirely. That split is the diagram headline.

The store behind the model

Multi turn apps add a store the diagram omits for clarity. Short term conversation history, a long term vector base for retrieval, and a tool result cache. The model reads and writes it on every turn, so eviction and freshness join the cost math.

Code

Route and cache with plain Python. The numbers match the worked example.

requests_per_day: int = 10000cache_rate: float = 0.30small_share: float = 0.70
cached: int = int(requests_per_day * cache_rate)remaining: int = requests_per_day - cachedsmall: int = int(remaining * small_share)large: int = remaining - small
cost_no_stack: int = requests_per_day * 10cost_stack: int = small * 1 + large * 10
print(f"{cached} {small} {large}")  # -> 3000 4900 2100print(f"{cost_no_stack} {cost_stack}")  # -> 100000 25900

The printout confirms the split and the cost fall. Flip cache rate to 0.00.0 and cost jumps to 37,00037,000. That gap is why cache comes before router tuning.

Watch Out For

A quiet layer fails while the model looks guilty

Accuracy dips on Tuesday. The model did not change. A guardrail threshold moved and now rejects 2020 percent of good answers, or a router flag sends everything large after a config edit.

Symptom: per layer counts shift while model only metrics stay flat, often with no error, just slower or shorter answers. Fix: require every layer to log with the same request id, alert on per layer reject and route shares from model monitoring, and prove the stack with testing ML systems before blaming weights.

Semantic cache reuses an answer it should not

Exact cache is safe and narrow. Semantic cache answers a new prompt with an old one at similarity 0.910.91, and the old answer carries yesterday's number.

Symptom: correct looking answers with stale entities, dates, or prices, clustered near paraphrases. Fix: keep exact and prefix cache broad, gate semantic reuse to stable intents with a high threshold and short TTL, and never cache personalised or time sensitive prompts.

The Quick Version

  • A production LLM app adds gateway, cache, router, guardrails, and an observability bus around inference.
  • Our 10,000 daily requests split into 3,000 cache hits, 4,900 small calls, and 2,100 large calls.
  • That split costs 25,900 units against 100,000 with no stack, about a 74 percent cut.
  • Gateway owns auth and limits, cache owns repeats, router owns model choice, guardrails own safety.
  • Every layer must emit spans with one shared request id or silent layer failures stay invisible.