Cache LLM Calls to Slash Latency and Cost in 2026
Every call to a large language model costs money and time. When your app scales to thousands of users, those milliseconds and micro-dollars compound into real latency problems and eye-watering invoices. LLM caching is the highest-leverage optimization most teams overlook — and in 2026, with strong open-source tooling and mature vector databases, there's no excuse to skip it.
This post walks through the two core caching patterns — exact prompt caching and semantic caching — plus the design decisions that separate a toy demo from a production-grade system.
Why Caching Matters More Than Ever
Modern applications frequently send near-identical prompts to an LLM. Customer support bots answer the same questions. RAG pipelines re-embed and re-summarize overlapping documents. Agents loop through repetitive planning steps. Without caching, you pay full price — in tokens and round-trip latency — for answers you've already generated.
The benefits stack up quickly:
- Lower cost: A cache hit costs a fraction of a full model inference, or nothing at all.
- Faster responses: Returning a cached answer from memory or a vector store typically takes milliseconds versus seconds for generation.
- Reduced rate-limit pressure: Fewer API calls means you stay comfortably under provider quotas.
- More consistency: Repeated questions get repeatable answers, which matters for compliance and QA.
Pattern 1: Exact Prompt Caching
The simplest pattern is a key-value lookup. You hash the full prompt (plus relevant parameters like model name, temperature, and system prompt) and store the response against that key.
How it works
Before calling the model, compute a deterministic hash of the request. Check your cache — Redis, Memcached, or even a local dictionary for small workloads. If the key exists, return the stored response. If not, call the LLM, store the result, and return it.
Here's the conceptual flow in Python:
import hashlib, json, redis
r = redis.Redis()
def cache_key(prompt, model, temperature):
payload = json.dumps({"p": prompt, "m": model, "t": temperature}, sort_keys=True)
return "llm:" + hashlib.sha256(payload.encode()).hexdigest()
def cached_completion(prompt, model, temperature, call_llm):
key = cache_key(prompt, model, temperature)
hit = r.get(key)
if hit:
return json.loads(hit)
result = call_llm(prompt, model, temperature)
r.set(key, json.dumps(result), ex=86400) # 24h TTL
return resultWhen to use it
Exact caching shines for deterministic, high-repeat workloads: FAQ answers, fixed system prompts with templated inputs, or batch jobs that reprocess the same data. Set temperature to 0 for cacheable endpoints so identical inputs reliably produce identical, reusable outputs.
The limitation
Exact caching is brittle. "How do I reset my password?" and "I forgot my password, how do I change it?" produce different hashes but deserve the same answer. That's where semantic caching earns its keep.
Pattern 2: Semantic Caching
Semantic caching matches requests by meaning rather than exact text. Instead of hashing, you embed the prompt into a vector, search for similar cached vectors, and return the stored answer if similarity crosses a threshold.
How it works
- Embed the incoming prompt using an embedding model.
- Query your vector store for the nearest cached prompt embedding.
- If cosine similarity exceeds your threshold (commonly 0.85–0.95), return the cached response.
- Otherwise, call the LLM, embed the new prompt, and store the pair.
Vector databases like Redis with vector search, Qdrant, Weaviate, or pgvector on Postgres all handle this well. Libraries such as GPTCache abstract much of the plumbing if you'd rather not build from scratch.
Tuning the similarity threshold
This is the most important knob. Set it too low and you'll serve wrong answers to genuinely different questions — a false positive that erodes trust. Set it too high and you barely beat exact caching. Start conservative around 0.92, then measure hit rate and error rate on real traffic before loosening it. Always log near-miss decisions so you can audit borderline matches.
Design Decisions That Make or Break Your Cache
Cache invalidation and TTLs
Stale answers are worse than slow ones. If your underlying knowledge base changes, cached responses referencing old data become liabilities. Use time-to-live expirations tuned to how fast your content changes, and invalidate cache entries when their source documents update. For RAG systems, tie cache keys to a version identifier of the retrieved context.
Scope keys correctly
A cached answer for one user may be inappropriate for another. If responses depend on user permissions, language, or region, include those attributes in the cache key. Never leak one tenant's data to another through a shared cache — namespace by tenant in multi-tenant apps.
Cache the right layer
You can cache at multiple points: the raw embedding, the retrieved chunks, or the final generated response. Embedding caching alone can cut costs dramatically in RAG pipelines since documents rarely change. Consider a layered approach rather than caching only the final output.
Handle streaming and partial responses
If your app streams tokens, decide whether to cache the complete assembled response and replay it, or skip caching for streamed endpoints. Most teams cache the full response and stream it back from cache on hits to preserve the UX.
Don't Forget Provider-Side Prompt Caching
Beyond your own cache, major providers now offer built-in prompt caching that discounts repeated prefix tokens — useful when you send a large static system prompt or shared context on every request. This is complementary to application-level caching: provider caching reduces the cost of the tokens you do send, while your semantic cache eliminates calls entirely. Structure prompts so static content comes first to maximize prefix reuse.
Measuring Success
Instrument your cache from day one. Track:
- Hit rate: the percentage of requests served from cache.
- Cost saved: estimated token spend avoided per hit.
- Latency delta: median response time for hits versus misses.
- False positive rate: semantic matches that returned an incorrect answer, ideally caught via human review or automated evals.
A well-tuned semantic cache on a repetitive workload can hit rates of 30–60%, translating directly into proportional cost and latency reductions. Start with exact caching to capture easy wins, layer in semantic matching where meaning varies, and guard everything with sensible TTLs and scoping. Your users get faster answers, and your finance team gets a smaller bill.
Ready to build real AI skills? Join the September 2026 cohort at Class For Jobs. Explore Advanced AI — a hands-on, live program to build and ship production AI applications, live and instructor-led with career support, resume help, and job-placement assistance.
Related reading
