Semantic Caching for LLM Apps: Cut Latency and API Costs

What semantic caching is and when it pays off

A regular cache needs an exact string match, and users rarely type the same string twice. "How do I reset my password?" and "forgot password, what now" are different strings with the same intent.

A semantic cache compares questions by meaning. Every question is turned into an embedding (a vector), and if a new question's vector is close enough to a stored one, you return the stored answer without calling the model.

It pays off when:

• many users ask similar questions (support bots, FAQ assistants, internal knowledge bases);
• answers are the same for everyone and change slowly;
• model calls are slow or expensive compared to an embedding call plus a vector lookup.

If every request is unique (code generation for a specific file, long personal conversations), the hit rate will be low and the cache only adds latency.

How it works step by step

1. Normalize the question: trim whitespace, unify case, strip greetings if they do not change meaning.
2. Embed it with the same embedding model you used for stored entries.
3. Search the nearest neighbour in a vector store, filtered by scope (see the cache key below).
4. If similarity is above the threshold, return the cached answer.
5. Otherwise call the LLM, return the answer and store the pair (vector, question, answer, metadata, timestamp).

A minimal lookup with cosine similarity on normalized vectors:

import numpy as np

def lookup(query_vec, entries, threshold):
    # query_vec and entry["vec"] are L2-normalized
    best, best_score = None, -1.0
    for entry in entries:
        score = float(np.dot(query_vec, entry["vec"]))
        if score > best_score:
            best, best_score = entry, score
    return best["answer"] if best and best_score >= threshold else None

In production a vector database index replaces the loop; the logic stays the same.

Choosing the similarity threshold

There is no universal number: it depends on the embedding model, the language and the domain. Calibrate it on your own data.

• Collect real question pairs from logs and label them: "same answer is correct" or "different answer needed".
• Pick a threshold that keeps false hits (wrong answer returned) close to zero, even if that lowers the hit rate. A missed hit costs one model call; a false hit costs user trust.
• Pay special attention to near-identical questions with different details: "refund for order 1452" vs "refund for order 1453", "tariff for 2 users" vs "for 20 users". Embeddings often place them very close.

A practical pattern is two thresholds: above the upper one return the cached answer, between the two ask a cheap model or a reranker to confirm the match, below the lower one call the main model.

The cache key is more than a vector

Similar meaning is not enough: an entry is valid only under the conditions it was created in. Filter by:

• model and system prompt version — a new prompt can change the correct answer;
• language of the answer;
• tenant, role or access level — a manager and a client may get different facts;
• source data version for RAG answers (index or document version).

Invalidation and staleness

Cached answers age. Plan for it early:

• TTL per category: long for stable reference content, short for prices, schedules or stock.
• Event-based invalidation: when a document in the knowledge base changes, delete entries that were built from it. For this, store the IDs of source documents with each entry.
• Version bump: changing the prompt or model starts a new namespace instead of purging everything manually.
• Feedback loop: a thumbs-down or a "this is wrong" report removes the entry and flags it for review.

When caching is unsafe

Do not serve cached answers when:

• the answer contains personal data or depends on the user's account, history or permissions;
• the answer depends on real-time state: balances, order status, availability;
• the request triggers actions through tools (booking, payment, sending messages) — cache the knowledge, never the side effect;
• legal, medical or financial wording must be checked per case.

The worst failure is a cross-user leak: user B gets an answer generated for user A. Scope the cache by tenant and user, or exclude personalized flows entirely.

Common mistakes

• Caching the whole conversation turn instead of standalone questions: a follow-up like "and how much is it?" means nothing without context.
• Mixing vectors from different embedding models in one index.
• Setting the threshold by feel and never measuring false hits.
• No way to purge entries after a content update.