A semantic cache for LLM calls with MongoDB Vector Search and Automated Embedding, in about 60 lines of Python.
Users ask the same questions in different words. "What is your refund policy?" and "How do refunds work?" are the same request, but an exact-match cache sees two different strings and pays for two LLM calls. A semantic cache fixes this by matching on meaning instead of text.
In this post I build one on MongoDB. The new part is Automated Embedding: MongoDB creates the embeddings for you, so there is no separate embedding pipeline to run. I will also show the guard that decides what should never be cached, because that part matters more than the cache itself.
Why an exact-match cache is not enough
A normal cache keys on the exact prompt string. That works for repeated system calls, but not for people. Small changes in wording, spacing or word order all count as a miss.
A semantic cache stores each prompt with its answer. For a new prompt it runs a vector search over the stored prompts. If the closest one is similar enough, it returns the stored answer and skips the LLM call. If not, it calls the LLM and stores the new pair.
Most of the work is in two decisions: which prompts are safe to cache, and how similar "similar enough" is.
Let MongoDB do the embedding
On May 7, 2026, MongoDB announced Automated Voyage AI Embeddings in Vector Search as a public preview. You declare an autoEmbed field in the index, and MongoDB embeds the text when documents are written and when you query. Here is the index for the cache collection, using PyMongo 4.20:
from pymongo.operations import SearchIndexModel INDEX = "prompt_cache_idx" def create_indexes(coll): model = SearchIndexModel( name=INDEX, type="vectorSearch", definition={"fields": [ {"type": "autoEmbed", "modality": "text", "path": "prompt", "model": "voyage-4"}, {"type": "filter", "path": "tenant"}, {"type": "filter", "path": "llm_model"}, ]}, ) coll.create_search_index(model=model) # plain TTL index: cached answers expire after 1 day coll.create_index("created_at", expireAfterSeconds=86_400) Two details matter here. The tenant and llm_model filter fields keep one customer's answers away from another's, and stop an answer from an old model being served after you switch models. The TTL index is a normal MongoDB feature: documents are deleted automatically a day after created_at, so stale answers do not live forever.
The lookup
With autoEmbed, the query takes plain text instead of a vector. I ask for the single nearest prompt and read its score:
THRESHOLD = 0.95 # illustrative, tune on your own traffic def cache_lookup(coll, prompt, tenant, llm_model): pipeline = [ {"$vectorSearch": { "index": INDEX, "path": "prompt", "query": {"text": prompt}, "filter": {"tenant": tenant, "llm_model": llm_model}, "numCandidates": 50, "limit": 1, }}, {"$project": {"answer": 1, "prompt": 1, "score": {"$meta": "vectorSearchScore"}}}, ] hit = next(coll.aggregate(pipeline), None) if hit and hit["score"] >= THRESHOLD: return hit return None vectorSearchScore is between 0 and 1, where 1 means most similar. The value 0.95 is only a starting point. The right number depends on your model and your users, so log the score of every lookup and look at real near-misses before you trust any threshold.
Not every prompt should be cached
This is where a semantic cache can hurt you. "Cancel order 48213" and "Cancel order 48214" are almost identical as text, but returning the first answer for the second request is a real bug. The same goes for anything personal or time-sensitive.
So I put a simple guard in front of the cache:
import re UNSAFE = re.compile( r"\d{3,}|@|\b(today|now|latest|current)\b", re.I) def is_cacheable(prompt: str) -> bool: """Skip prompts with IDs, emails or time words.""" return not UNSAFE.search(prompt) I ran this guard locally on six sample prompts. My first version also blocked the words "my" and "me", and it rejected "How do I reset my password?", which is one of the most cacheable questions there is. After removing those words, the run gave: password reset, refund policy and file formats are cacheable; the order number, the email address and "latest prices" are skipped. A regex is a blunt tool, and in a real system you would also check the route or intent, but the point stands: decide what to exclude before you decide the threshold.
The full flow then checks the guard on both read and write:
from datetime import datetime, timezone def answer(coll, prompt, tenant, llm_model, call_llm): if is_cacheable(prompt): hit = cache_lookup(coll, prompt, tenant, llm_model) if hit: return hit["answer"], "cache" text = call_llm(prompt) if is_cacheable(prompt): coll.insert_one({ "prompt": prompt, "answer": text, "tenant": tenant, "llm_model": llm_model, "created_at": datetime.now(timezone.utc), }) return text, "llm" Before you ship it
- Automated Embedding is a preview feature, and MongoDB's docs say not to use it in production yet. Check the current prerequisites for your cluster tier before you build on it.
- Embedding calls are billed, both when you write and when you query. A cache saves LLM tokens but adds embedding cost, so measure both.
- When your source documents change, cached answers can go stale before the TTL ends. Add a version field to the filter, or clear the collection on deploy.
- Return the cache source ("cache" or "llm") in your logs. Without it you cannot measure the hit rate or debug a wrong answer.
I only ran the pure Python parts here (the guard and the index model). The vector search calls need an Atlas cluster with Automated Embedding enabled.
Takeaway
A semantic cache is a small amount of code: one index, one $vectorSearch stage and one threshold. The hard part is deciding what must never be cached. Scope by tenant and model, block IDs and time-sensitive prompts, give everything a TTL, and tune the threshold on real logs, not on a guess.
Sources
- How to index fields for vector search (autoEmbed) – MongoDB Docs
- $vectorSearch aggregation stage – MongoDB Docs
- The Bottleneck in Enterprise AI Isn't the Model. It's the Data – MongoDB Blog
- TTL indexes – MongoDB Manual
- PyMongo on PyPI
Let's Connect
I write about AI agents, .NET, and building on MongoDB every week. If this post helped you, let's connect:
Follow me here for the next post!
