Core ML

LLM Caching

LLM inference is expensive. Caching is how you make some requests free. The trick is knowing which of the three caching strategies to use, because each one saves money in a different situation — and each one has a failure mode the others don't.

Why Does This Exist?

A call to GPT-4o costs roughly $\$ 5$ per million output tokens. If your app has 10,000 users a day each sending similar support questions, you're paying for the same computation thousands of times. Caching intercepts those repeated requests and returns a stored answer instead of running inference again.

But LLMs aren't like web pages. Two requests that mean the same thing are almost never character-for-character identical. This is why LLM caching is harder than HTTP caching, and why three distinct strategies exist to handle different situations.

Think of It Like This

Imagine a very expensive consultant who charges $\$ 500$ per question. You start writing down every question-answer pair. For exact repeats, you hand back the written answer for free. For questions about the same contract (shared context), you give them the contract upfront so they don't re-read it each time. And for questions that are clearly just rephrasings of old ones, you check your notes to see if you already have a close enough answer. Three different strategies, all serving the same goal: pay the consultant as rarely as possible.

Three Caching Strategies

1. Exact Match

Hash the full prompt string. On a match, return the cached response. This is the fastest and cheapest check — a single key lookup in Redis takes under a millisecond.

When it works well: FAQ bots, template-driven apps where the prompt is assembled from fixed parts and a small set of variable slots, or any system where the same literal prompt can appear repeatedly (like a code review bot on a popular open-source library).

Where it fails: Real user-typed queries. People type "what's the return policy?" and "can I return my order?" to mean the same thing. The hashes differ. Exact match misses both.

2. Prefix (KV Cache) Reuse

LLMs build a KV cache internally as they process tokens. If two requests start with the same system prompt and the model server is aware of this, it can reuse the KV computations for the shared prefix and only process the new user turn from scratch. OpenAI's API does this automatically for long prompts that share a prefix; vLLM supports it natively for self-hosted models.

When it works well: Apps where every user sees the same system prompt — a customer support bot with a 2,000-token instructions block, or a coding assistant with a long repository context. The savings scale linearly with the shared prefix length.

Where it fails: If your system prompt is different for every user (personalised), there's no shared prefix and no savings.

3. Semantic Match

Embed the incoming query using a small embedding model. Search a vector database for the nearest past query. If the cosine similarity is above a threshold (typically 0.95), return the cached answer.

When it works well: Support bots where users ask the same question in many phrasings. A single "what's your cancellation policy" answer covers "how do I cancel?", "can I get out of my subscription?", and "cancellation fee?" all in one cache entry.

Where it fails: The embedding step adds 5–20ms of latency. False positives — queries that are semantically close but need different answers — will serve wrong responses silently. The threshold tuning is finicky. And for highly dynamic or personalised content, every query is unique, so the cache hit rate stays near zero.

Show Me the Code

A simple semantic cache using sentence-transformers and FAISS:

1import numpy as np2import faiss3from sentence_transformers import SentenceTransformer4
5class SemanticCache:6    def __init__(self, threshold: float = 0.95):7        self.model = SentenceTransformer("all-MiniLM-L6-v2")8        self.threshold = threshold9        self.index = faiss.IndexFlatIP(384)  # inner product = cosine on normalised vecs10        self.answers: list[str] = []11
12    def _embed(self, text: str) -> np.ndarray:13        vec = self.model.encode([text], normalize_embeddings=True)14        return vec.astype("float32")15
16    def get(self, query: str) -> str | None:17        if self.index.ntotal == 0:18            return None19        vec = self._embed(query)20        scores, ids = self.index.search(vec, 1)21        if scores[0][0] >= self.threshold:22            return self.answers[ids[0][0]]23        return None24
25    def set(self, query: str, answer: str) -> None:26        vec = self._embed(query)27        self.index.add(vec)28        self.answers.append(answer)29
30cache = SemanticCache(threshold=0.95)31
32def call_with_cache(query: str) -> str:33    cached = cache.get(query)34    if cached:35        return cached  # free36    answer = call_llm(query)  # full inference cost37    cache.set(query, answer)38    return answer

Watch Out For

Stale cache poisoning. LLM caches have no automatic invalidation mechanism. If your product policies change, your cached answers are silently wrong until their TTL expires — or forever, if you set no TTL. Always set a TTL (24 hours is common) and flush the cache on any prompt or knowledge-base update. The semantic cache is especially risky here: a wrong answer cached under embedding vector A will silently serve itself for every semantically-similar query B, C, and D.

The Quick Version

Exact match is fastest but only saves money on truly identical prompts.
Prefix (KV) reuse is transparent — the model server does it automatically — and works whenever requests share a long common prefix.
Semantic match handles rephrased queries at the cost of embedding latency and a false-positive risk.
Layer all three: exact first (cheapest check), then prefix (free if the model server supports it), then semantic as a last resort.
Always set a TTL. A stale cache that returns wrong answers costs more than no cache at all.