---
title: Your Marketing AI Answers the Same Question 400 Times a Day. Cache It Semantically.
description: Exact-match caching misses nearly every repeat question because people phrase things differently. A semantic cache matches on meaning, cuts model spend on repeat traffic, and answers in milliseconds.
author: LETSGROW Dev Team
date: 2026-11-04
category: AI Tools
tags: ["Semantic Caching", "LLM", "Embeddings", "Marketing AI", "Cost Optimization"]
url: "https://letsgrow.dev/blog/semantic-caching-llm-marketing-ai-repeat-queries"
---
# Your Marketing AI Answers the Same Question 400 Times a Day. Cache It Semantically.

Look at the prompts your marketing AI handles in a week and a pattern shows up fast. "What is our refund policy?" "How do I connect HubSpot?" "Summarize this campaign for the weekly report." The wording changes, the question does not. You are paying a model to rederive the same answer, with the same latency, hundreds of times.

The position: any customer-facing or internal marketing assistant with repeat traffic should sit behind a semantic cache. Exact-match caching misses almost everything because humans never phrase a question the same way twice. A semantic cache matches on meaning, returns a stored answer in milliseconds, and cuts your model bill on repeat traffic without touching quality when you tune it properly.

## Why Prompt Caching Is Not Enough

Prompt caching, offered by the major model providers, discounts the repeated prefix of a prompt, such as a long system message. It still runs a full generation every time. You pay for the output, you wait for the output, and the answer can drift between calls.

A semantic cache sits in front of the model. It embeds the incoming query, looks for a stored query within a similarity threshold, and returns the stored answer if it finds one. No model call, no generation cost, no drift. The two techniques stack: prompt caching for the misses, semantic caching for the hits.

::compare-table
title: "Three Caching Layers for Marketing AI"
columns: ["Layer", "What it matches", "What you save", "Risk"]
rows:
  - ["Exact-match cache", "Identical strings", "Full call, rare hits", "Near zero, but hit rate is tiny"]
  - ["Prompt caching", "Repeated prompt prefix", "Discounted input tokens", "Low, output still generated"]
  - ["Semantic cache", "Similar meaning", "Entire call, high hit rate", "Wrong match if threshold is loose"]
::

## Build It in an Afternoon

The mechanics are small. Embed the query with a cheap embedding model. Search a vector index for the nearest stored query. If cosine similarity clears your threshold, return the cached answer. Otherwise call the model, then store the new query, its embedding, and the answer.

```python
def answer(query, scope):
    vec = embed(query)
    hit = index.search(vec, filter={"scope": scope}, top_k=1)
    if hit and hit.score >= THRESHOLD and not expired(hit):
        return hit.answer
    result = llm(query)
    index.upsert(vec, result, scope=scope, ttl=TTL)
    return result
```

Three details decide whether this saves money or ships wrong answers. First, the scope filter. A cached answer about pricing for one product line or one customer tier must never serve another. Put product, locale, and audience in the cache key, not just the question text. Second, the TTL. Pricing, promotions, and policy answers expire when the source changes, so tie expiry to your content publish events, not a fixed clock. Third, personalization. Never cache a response that contains user-specific data. If the answer depends on who is asking, it does not belong in a shared cache.

## Tune the Threshold With Your Own Data

Everyone asks for the magic similarity number. There isn't one. Embedding models score differently, and your domain has near-duplicates that mean opposite things, like "cancel my plan" and "upgrade my plan." Both sit close together in embedding space and need different answers.

Do this instead. Pull 500 real queries from your logs. Pair each with its nearest neighbor. Have someone label every pair as same answer or different answer. Plot the score distribution for both groups and pick the threshold where wrong matches fall below one percent. Start strict, usually well above 0.9 on normalized embeddings, and loosen only as the labeled data supports it.

::checklist
title: "Semantic Cache Launch Checklist"
items:
  - "Scope keys include product, locale, and audience tier"
  - "No user-specific or PII-bearing responses stored"
  - "TTL tied to content publish and pricing change events"
  - "Threshold set from 500 labeled query pairs, not a guess"
  - "Hit rate, false-hit rate, and saved spend logged per workflow"
  - "Manual purge endpoint for a bad cached answer"
::

## Measure the Two Numbers That Matter

Track hit rate and false-hit rate side by side. Hit rate tells you the savings. False-hit rate, the share of cache hits a reviewer would mark as wrong, tells you the cost. A cache at 60 percent hits with a 3 percent false-hit rate is a liability for a support assistant, even though the savings chart looks great. Sample ten hits a day for human review and alert if the false-hit rate climbs.

Add a shadow mode for the first week. Serve every answer from the model, but compute what the cache would have returned and compare. You get your real hit rate and a labeled dataset with zero customer risk.

## The Takeaway

Pick the one assistant with the most repeat traffic. Add scope keys, a content-driven TTL, and a strict threshold. Run shadow mode for a week, review the false hits, and then turn it on. Teams that do this routinely see a large share of FAQ-style traffic answered without a model call, with faster responses and a smaller bill. Caching is the cheapest quality and speed upgrade in your AI stack, and most marketing teams have not touched it.
