A semantic cache in Redis is the cheapest way to cut AI API costs in production, and it is also the easiest way to serve a confidently wrong answer to a customer who asked something slightly different. I have built both versions: the one that saved a company five figures a year, and the one that answered “how do I downgrade” with the enterprise upgrade instructions for three weeks before anyone noticed.

The difference was not the vector index or the embedding model. It was whether the cache had a measured threshold, a versioned key, and a stated error budget. This page covers the economics honestly, the way to choose a distance threshold from data instead of copying 0.9 from a blog post, the conditions that should bypass the cache entirely, the Redis schema and lookup code, and the metrics that tell you the cache is helping rather than hiding a correctness problem.

For the surrounding cost picture, pair this with LLM token cost optimization, the operational accounting in hidden costs of running AI in production, and the request-side controls in API rate limiting and cost management. If your retrieval layer already lives in Postgres, the same filtering logic applies to hybrid search in pgvector, and tenant scoping for cached answers is the same problem as RAG access control.

How Does Semantic Caching Reduce AI API Costs?

Every uncached request pays for the same computation twice: once as input tokens for the system prompt and retrieved context, once as output tokens for the answer. If a support inbox receives the same forty questions in forty phrasings, the provider bills forty times for one piece of work.

Caching attacks that duplication at three levels, and they are not interchangeable:

Layer Hit condition What it captures Main risk
Exact cache Byte-identical key Retries, double submits, refreshes Almost none
Normalized cache Lowercased, trimmed, punctuation and template stripped Copy-paste variations Slightly wrong normalization merges distinct questions
Semantic cache Embedding distance under a threshold Paraphrases and rewordings Serving an answer to a question nobody asked

The first two are safe and boring, which is why they should exist before the third. The semantic layer is where the savings are, and where the mistakes live.

The economics are simpler than most vendors make them sound:

savings = h × C − (e + m)

h is the true hit rate, C is the average cost of an uncached answer, e is the cost of embedding each incoming query, and m is the amortized cost of the Redis instance and its memory. Because e and m are small in most deployments, savings are very close to the hit rate times the bill. A claim of “60 percent savings” is really a claim that 60 percent of your traffic is genuinely duplicate work.

Worked through with stated assumptions, for a product generating 20,000 model answers per day at an average of $0.004 per answer (roughly 2,000 input tokens and 400 output tokens on a mid-tier model):

  • Daily spend without a cache: $80.
  • At a 30 percent hit rate: $24 saved per day, about $730 per month.
  • At a 60 percent hit rate: $48 saved per day, about $1,460 per month.
  • Embedding cost, if you embed locally: effectively the CPU time of a small model. If you call a hosted embedding API for all 20,000 queries, you add one network hop and one API call per query, which changes both the latency and the break-even math.

Those numbers are assumptions, not quotes. They scale linearly with your traffic and your token bill, and the ratio is what matters: the cache is worth building when the duplicate share of your traffic is large relative to the daily allowance you are willing to spend on engineering and memory.

What Vector Distance Threshold Works Best for Redis Semantic Cache?

There is no universal number, and the single most common implementation error is treating one as if there were.

Start with the units. Redis cosine distance is 1 − cosine similarity, so it runs from 0 (identical direction) to 2 (opposite), and lower means stricter. The RedisVL documentation ships a default of distance_threshold = 0.1, which corresponds to roughly 0.9 cosine similarity, and notes that lower is stricter. That default is a reasonable starting point for a first experiment, not a decision.

The hard part is that a fixed similarity threshold does not bound the rate of wrong answers. Semantic Caches for LLM Web Services Leave Their Error Budget Unspent frames the problem the right way: a reused answer can be wrong, so the share of wrong reused answers must stay below a target error budget, and fixed thresholds tend to break that budget when the query stream shifts. The paper also reports that a learned per-entry approach realizes only 12 to 55 percent of the available error budget on three public query streams, which is a polite way of saying that even the research systems are leaving accuracy on the table. The practical takeaway is that thresholds are an optimization problem with a measured objective, and the objective is false hits, not hit rate.

The method I use, and the one the Redis threshold optimization guide describes for RedisVL, is a sweep against labeled data:

  1. Sample real traffic. Pull 1,000 to 5,000 recent queries from logs, weighted toward your highest-volume intents. Do not use synthetic paraphrase pairs; the distribution is the point.
  2. Build candidate pairs. For each sampled query, find its nearest cached neighbour at a loose threshold so you get plenty of borderline pairs.
  3. Label the pairs. A human decides whether the cached answer is acceptable for the new query. Acceptable means a customer would not notice the substitution, not “roughly related”.
  4. Sweep the threshold. For each candidate distance, compute the hit rate and the false-hit rate. Plot both.
  5. Pick the strictest point that keeps false hits under budget. If your budget is 0.5 percent of served answers, choose the threshold where measured false hits sit below that with margin, then re-run the sweep when the corpus, the embedding model, or the prompt template changes.

Directional starting points, to be treated as hypotheses rather than rules: below 0.05 distance for account, billing, and policy answers where a wrong answer costs money; 0.05 to 0.12 for documentation and product search where a near-miss is survivable; anything above roughly 0.2 requires evidence I have not seen in production systems.

Halftone newsprint dial with a dark half and a pale half and a needle resting near the dark side, representing a similarity threshold tradeoff

The failure this prevents is mundane. “How do I upgrade to enterprise?” and “How do I downgrade my plan?” sit close together in embedding space because they share almost every token, and they have opposite correct answers. A loose threshold merges them. No index tuning fixes that; only a measured cut-off does.

When Should Web Applications Bypass a Semantic Cache?

Some requests must never be served from cache, regardless of threshold. Treat this list as a policy layer that runs before the vector lookup, not as an optimization.

Condition Why the cache is unsafe What to key on instead
Personal or account-specific questions The nearest neighbour may be another user’s answer User ID plus session, or exclude entirely
Anything with a side effect Cached “cancel my subscription” responses hide a real action Never cache actions or tool calls
Live data Prices, stock, telemetry, and status change by the minute Time-bucketed keys with short TTL, or no cache
Answers grounded in a changed document The stored answer cites content that no longer exists Document version or index version in the key
Regulated data (health, financial, legal) Reuse can cross a consent boundary Tenant and consent scope in the key, or exclude
Creative or non-deterministic requests A repeated answer is a bug to the user Exclude by request type
High-impact or irreversible advice The cost of a wrong answer dwarfs the token saving Exclude, or serve with a fresh verification step
Evaluation and canary traffic Cached answers contaminate your quality metrics Bypass and tag the request

Two of these deserve emphasis because they are operational rather than architectural.

Version the key, not just the threshold. A cached answer is only correct for the model, prompt template, and knowledge state that produced it. Include a hash of the system prompt, the model identifier, and the retrieval index version in the cache key or in a metadata field you filter on. When any of them changes, bump the version and the old entries stop matching. Without this, a prompt fix ships and the cache keeps serving the pre-fix answers until TTL expires.

Halftone newsprint key tag with three torn strips stapled along its length, representing a cache key versioned by tenant, model, and prompt

Cached answers are an injection surface. If an attacker can get a crafted answer stored, later users with a similar query receive it. That is the same trust problem as prompt injection in retrieved content, with a longer shelf life. Only cache answers that passed whatever validation you already run, keep TTLs short for anything user-influenced, and never cache an answer that includes tool output from an external source without a version stamp.

How Do You Build the Cache in Redis?

Halftone newsprint illustration of a request passing through a wheel into a stack of filed sheets and splitting into a filled slot and an empty slot

Redis keeps the whole cache in the same process as your other fast data, which is why it wins here: one network hop, one operational surface, sub-millisecond index lookups on top of an embedding step that usually dominates the latency budget.

The index needs a vector field plus the metadata you filter and version on:

import redis
from redis.commands.search.field import TextField, NumericField, VectorField
from redis.commands.search.index_definition import IndexDefinition, IndexType

def create_cache_index(client: redis.Redis, index_name: str = "idx_llm_cache", dims: int = 1536):
    schema = (
        TextField("prompt_hash"),
        TextField("tenant_id"),
        TextField("prompt_version"),
        TextField("model_id"),
        NumericField("created_at"),
        VectorField(
            "prompt_vector",
            "HNSW",
            {"TYPE": "FLOAT32", "DIM": dims, "DISTANCE_METRIC": "COSINE", "M": 16, "EF_CONSTRUCTION": 200},
        ),
    )
    definition = IndexDefinition(prefix=["cache:"], index_type=IndexType.HASH)
    client.ft(index_name).create_index(schema, definition=definition)

The lookup does a tenant-filtered nearest neighbour search and then applies the threshold in application code, which keeps the decision visible and testable:

import numpy as np
import redis
from redis.commands.search.query import Query

class SemanticCache:
    def __init__(self, client: redis.Redis, index="idx_llm_cache", threshold=0.05):
        self.client, self.index, self.threshold = client, index, threshold

    def lookup(self, vector: list[float], tenant_id: str, prompt_version: str):
        q = (
            Query(f"(@tenant_id:{{{tenant_id}}} @prompt_version:{{{prompt_version}}})=>[KNN 3 @prompt_vector $vec AS dist]")
            .sort_by("dist")
            .return_fields("response", "dist", "created_at")
            .dialect(2)
        )
        params = {"vec": np.asarray(vector, dtype=np.float32).tobytes()}
        docs = self.client.ft(self.index).search(q, query_params=params).docs
        if not docs:
            return None
        best = docs[0]
        distance = float(best.dist)
        return {"response": best.response, "distance": distance} if distance <= self.threshold else None

Three implementation notes that matter more than the code:

  • Embed locally if you can. Redis’s own semantic cache tooling ships with a local embedding model by default (redis/langcache-embed-v2), and the reason is latency and cost: a hosted embedding call per query adds a round trip and re-introduces exactly the dependency the cache was meant to reduce. A small local model adds single-digit milliseconds and no per-query bill.
  • Use RedisVL if you do not want to own the plumbing. Its SemanticCache class wraps index creation, store, check, TTL, and filter expressions, and its defaults are documented rather than hidden. Writing the raw commands is fine when you need control over key composition.
  • Set TTLs by answer volatility. Documentation answers can live for days. Policy answers tied to a versioned document should expire with that document. Account answers should not be cached at all.

What Does the Cache Cost to Run?

Memory is the recurring cost, and the arithmetic is unforgiving enough to plan for.

A 1,536-dimension float32 embedding is 6,144 bytes before metadata. Add the hash fields, the key name, and HNSW graph overhead, and plan on roughly 7 KB per entry. That is about 0.7 GB for 100,000 entries and about 7 GB for a million. Halftone newsprint illustration of a fixed bin packed with paper slips and a few spilling over the rim, representing a cache memory budget

Smaller embedding models change the math linearly: 768 dimensions halves it, and quantization options cut it further at some accuracy cost.

Two operational rules keep the cache from becoming an incident:

  • Give it a dedicated instance or logical database. Sharing with session data or rate-limit counters means an eviction policy tuned for one workload silently corrupts the other. allkeys-lru on a shared Redis will evict your session keys when the cache fills.
  • Plan for cold starts. A wiped cache means a spike of uncached requests, which is both a cost spike and a latency spike. Warm the top intents from logs at deploy time if the provider bill is sensitive to bursts.

How Do You Prove It Works and Is Not Just Hitting?

A cache with a 70 percent hit rate that quietly serves wrong answers is a liability with good dashboards. The metrics that matter are paired:

  • Hit rate and false-hit rate. Sample 100 served hits a week and have a human label whether the answer was acceptable. If false hits exceed your budget, tighten the threshold before you celebrate the savings.
  • Cost per correct answer. Total spend divided by answers a user found useful. This catches a cache that saves tokens while increasing support contacts.
  • p50 and p95 latency end to end. Include the embedding step and the Redis lookup. A cache hit that still takes 300 ms because of a hosted embedding call is not the 20 ms story the marketing implies.
  • Bypass rate. The share of traffic excluded by policy. If it is 60 percent, your savings ceiling is 40 percent.
  • Staleness incidents. Count cases where a cached answer was demonstrably out of date. One per month is a process failure, not noise.

Run the cache in shadow mode first: compute the hit and the live answer, serve the live one, and compare offline. Two weeks of shadow data tells you the real hit rate and the real false-hit rate before a single customer sees a cached answer.

Five Failure Modes I See

Symptom Real cause Fix
Wrong answers after a prompt or model change Key ignores prompt template and model version Version those fields and bump on deploy
Cache hit rate collapses overnight Index rebuild or eviction dropped entries Warm top intents, separate instance, monitor eviction
Users report stale policies TTL not tied to document version Version the key on the source document
Latency barely improves Hosted embedding call per query Embed locally or pre-embed at the edge
Savings never materialize Traffic is not actually duplicate Measure duplicate share before building

Conclusion

Semantic caching is a cost lever with a correctness bill attached, and most of the engineering work is in the second part. The fixed threshold that saves the most money is also the one most likely to answer a question nobody asked, so pick it from labeled pairs and a stated error budget rather than from a blog post. Keep exact and normalized caches underneath it, version the key on everything that changes an answer, bypass for anything with a side effect, and prove the hit rate and the false-hit rate together before the cache is on the critical path.

Done that way, the cache is one of the few AI cost reductions that does not degrade the product. Done the other way, it is a machine for confidently repeating yesterday’s answer to today’s question.

AI Architecture & Cloud FinOps

Ready to cut your AI infrastructure and API bills?

Veduis helps software startups, agencies, and web developers design Redis caching layers, vector search architectures, and resilient AI backends.