If you run an LLM-backed endpoint (a support bot, a docs assistant, a "chat with your data" feature), you've probably noticed something in your logs: people ask the same questions over and over, just worded differently.

  • "How do I reset my password?"
  • "I forgot my password, what do I do?"
  • "password reset steps"

A traditional cache keyed on the exact prompt string treats these as three unrelated requests, so you pay for three LLM calls, three times the latency, and three times the tokens.

A semantic cache fixes this. Instead of matching strings, it matches meaning: embed the incoming prompt, find the nearest previously answered prompt, and if it's close enough, return the stored answer without ever calling the LLM.

In this post we'll build one in Node.js and TypeScript with a two-tier design:

  • L1: in-process memory. Microsecond lookups for exact repeats.
  • L2: Postgres + pgvector. Shared across instances, durable, with similarity search via an HNSW index.

We'll also cover the part most tutorials skip: when a cache hit is actually a wrong answer, and how to design around it.

How a semantic cache works 

Request ─▶ normalize ─▶ L1 exact lookup ──hit──▶ return │ miss ▼ embed(prompt) ◀── ~50-300 ms, small cost │ ▼ L2 nearest neighbor (pgvector) │ similarity ≥ threshold? ──yes──▶ return cached answer (+ promote to L1) │ no ▼ call the LLM │ ▼ store in L1 + L2 (embedding + answer) ─▶ return

The economics work because embedding a prompt is much cheaper and faster than generating a completion. A miss costs you one extra embedding call plus a DB query. A hit saves a full generation. If your hit rate is decent, you come out well ahead on both cost and p50 latency.

Real numbers depend entirely on your traffic, so measure them (we'll add metrics later) rather than trusting anyone's blog post, including this one.

Step 1: Set up pgvector

Enable the extension and create the cache table:

CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE llm_cache ( id BIGSERIAL PRIMARY KEY, scope TEXT NOT NULL, -- tenant + model + prompt version (see below) prompt_hash TEXT NOT NULL, -- hash of normalized prompt, for exact match prompt TEXT NOT NULL, -- original text, useful for debugging and audits embedding vector(1536) NOT NULL, response JSONB NOT NULL, hit_count INTEGER NOT NULL DEFAULT 0, created_at TIMESTAMPTZ NOT NULL DEFAULT now(), expires_at TIMESTAMPTZ NOT NULL ); -- Exact-match lookups CREATE UNIQUE INDEX llm_cache_scope_hash_idx ON llm_cache (scope, prompt_hash); -- Approximate nearest neighbor search using cosine distance CREATE INDEX llm_cache_embedding_idx ON llm_cache USING hnsw (embedding vector_cosine_ops); -- For cleanup jobs CREATE INDEX llm_cache_expires_idx ON llm_cache (expires_at);

Notes:

  • vector(1536) must match your embedding model's output dimension. Check your model's docs, and keep it consistent. Mixing models in one column breaks similarity.
  • HNSW gives fast approximate search and doesn't require training data first (unlike IVFFlat). It needs pgvector 0.5.0 or newer.
  • <=> is pgvector's cosine distance operator. Cosine similarity is 1 - distance.

The scope column is the most important design decision

A cached answer is only valid for the context it was generated in. Put everything that changes the correct answer into scope:

function buildScope(opts: { tenantId: string; model: string; promptVersion: string; // bump when you change your system prompt toolsetVersion?: string; }) { return [opts.tenantId, opts.model, opts.promptVersion, opts.toolsetVersion ?? '-'].join('|'); }

Without this, tenant A's answer can be served to tenant B (a data leak), or a stale answer generated by an old system prompt keeps being served after you "fixed" the bot. Bumping promptVersion is your cheap, instant "invalidate everything" button.

Step 2: The cache class

Install dependencies:

npm i pg pgvector lru-cache

Here's the core implementation:

// semantic-cache.ts import crypto from 'node:crypto'; import { Pool } from 'pg'; import pgvector from 'pgvector/pg'; import { LRUCache } from 'lru-cache'; export interface CachedResponse { text: string; model: string; usage?: { inputTokens: number; outputTokens: number }; } export interface CacheLookup { hit: boolean; tier?: 'l1' | 'l2'; similarity?: number; response?: CachedResponse; } export class SemanticCache { // L1: exact-match, in-process. Key = scope + prompt hash. private l1 = new LRUCache<string, CachedResponse>({ max: 5_000, ttl: 10 * 60 * 1000, // 10 minutes }); // Single-flight: if two identical requests arrive at once, only one calls the LLM private inflight = new Map<string, Promise<CachedResponse>>(); constructor( private pool: Pool, private embed: (text: string) => Promise<number[]>, private opts = { similarityThreshold: 0.92, ttlSeconds: 24 * 60 * 60, }, ) {} static normalize(prompt: string): string { return prompt.trim().toLowerCase().replace(/\s+/g, ' '); } private hash(text: string) { return crypto.createHash('sha256').update(text).digest('hex'); } async lookup(scope: string, prompt: string): Promise<CacheLookup & { embedding?: number[] }> { const normalized = SemanticCache.normalize(prompt); const promptHash = this.hash(normalized); const l1Key = `${scope}:${promptHash}`; // --- L1: exact match in memory --- const l1Hit = this.l1.get(l1Key); if (l1Hit) return { hit: true, tier: 'l1', similarity: 1, response: l1Hit }; // --- L2a: exact match in Postgres (no embedding needed) --- const exact = await this.pool.query( `SELECT response FROM llm_cache WHERE scope = $1 AND prompt_hash = $2 AND expires_at > now()`, [scope, promptHash], ); if (exact.rows[0]) { this.l1.set(l1Key, exact.rows[0].response); return { hit: true, tier: 'l2', similarity: 1, response: exact.rows[0].response }; } // --- L2b: semantic match via pgvector --- const embedding = await this.embed(normalized); const { rows } = await this.pool.query( `SELECT id, response, 1 - (embedding <=> $2) AS similarity FROM llm_cache WHERE scope = $1 AND expires_at > now() ORDER BY embedding <=> $2 LIMIT 1`, [scope, pgvector.toSql(embedding)], ); const best = rows[0]; if (best && best.similarity >= this.opts.similarityThreshold) { // Fire-and-forget stats update; don't block the response this.pool .query('UPDATE llm_cache SET hit_count = hit_count + 1 WHERE id = $1', [best.id]) .catch(() => {}); this.l1.set(l1Key, best.response); // promote so the exact phrasing is fast next time return { hit: true, tier: 'l2', similarity: best.similarity, response: best.response, embedding }; } return { hit: false, embedding }; } async store( scope: string, prompt: string, response: CachedResponse, embedding?: number[], ): Promise<void> { const normalized = SemanticCache.normalize(prompt); const promptHash = this.hash(normalized); this.l1.set(`${scope}:${promptHash}`, response); const vec = embedding ?? (await this.embed(normalized)); await this.pool.query( `INSERT INTO llm_cache (scope, prompt_hash, prompt, embedding, response, expires_at) VALUES ($1, $2, $3, $4, $5, now() + ($6 || ' seconds')::interval) ON CONFLICT (scope, prompt_hash) DO UPDATE SET response = EXCLUDED.response, embedding = EXCLUDED.embedding, expires_at = EXCLUDED.expires_at`, [scope, promptHash, prompt, pgvector.toSql(vec), response, String(this.opts.ttlSeconds)], ); } /** Wrap an LLM call with cache lookup, single-flight, and write-back. */ async getOrCompute( scope: string, prompt: string, compute: () => Promise<CachedResponse>, ): Promise<CacheLookup & { response: CachedResponse }> { const found = await this.lookup(scope, prompt); if (found.hit) return found as CacheLookup & { response: CachedResponse }; const key = `${scope}:${this.hash(SemanticCache.normalize(prompt))}`; let pending = this.inflight.get(key); if (!pending) { pending = compute() .then(async (res) => { // Don't let a cache write failure fail the user's request await this.store(scope, prompt, res, found.embedding).catch((e) => console.error('cache store failed', e), ); return res; }) .finally(() => this.inflight.delete(key)); this.inflight.set(key, pending); } return { hit: false, response: await pending }; } }

A few things worth calling out:

  • Exact match before embedding. Many repeats are literally identical. Skipping the embedding call for those saves both time and money.
  • Single-flight. If 50 users ask the same new question at once (say, right after an outage), you want one LLM call, not 50. The inflight map handles that within a process.
  • The cache must never break the request. Failures on store are logged and swallowed. You should also wrap lookup so a database outage degrades to "just call the LLM" (see the Express example below).
  • Parameterized queries throughout. pgvector.toSql() serializes the array safely for the vector type.

Step 3: Wire it into an endpoint 

// server.ts import express from 'express'; import { Pool } from 'pg'; import pgvector from 'pgvector/pg'; const pool = new Pool({ connectionString: process.env.DATABASE_URL }); pool.on('connect', (client) => pgvector.registerTypes(client)); const cache = new SemanticCache(pool, embedText); const app = express(); app.use(express.json()); app.post('/api/ask', async (req, res) => { const { question } = req.body; const tenantId = req.user.tenantId; // from your auth middleware const scope = buildScope({ tenantId, model: process.env.LLM_MODEL!, promptVersion: 'support-bot-v7', }); const started = performance.now(); try { const result = await cache.getOrCompute(scope, question, () => callLlm({ system: SYSTEM_PROMPT, user: question }), ); metrics.timing('ask.latency_ms', performance.now() - started, { cache: result.hit ? result.tier! : 'miss', }); metrics.increment('ask.cache', { result: result.hit ? 'hit' : 'miss' }); res.json({ answer: result.response.text, cached: result.hit }); } catch (err) { // If the cache layer itself blows up, fall back to the LLM directly console.error('cache path failed, calling LLM directly', err); const direct = await callLlm({ system: SYSTEM_PROMPT, user: question }); res.json({ answer: direct.text, cached: false }); } });

One caveat on that catch: it also catches errors from the LLM call itself, so you may double-call the LLM if the first call failed. In production, separate "cache infrastructure errors" from "compute errors" (for example by having lookup return a miss on DB failure internally).

The hard part: choosing a similarity threshold

The threshold is the knob that trades hit rate against correctness, and there's no universal right value. It depends on your embedding model, your domain, and how costly a wrong answer is.

  • Too low: you serve wrong answers confidently.
  • Too high: you rarely hit, and the cache is expensive overhead.

Embedding similarity measures topical closeness, not intent equivalence. These pairs are all highly similar to an embedding model but need different answers:

Prompt A Prompt B Same answer OK?
"How do I cancel my subscription?" "How do I not cancel my subscription?" ❌
"What's the refund policy for annual plans?" "What's the refund policy for monthly plans?" ❌
"Is feature X available in Python?" "Is feature X available in Go?" ❌
"How do I reset my password?" "I forgot my password" ✅

Calibrate with your own data

Don't guess. Build a small labeled set from real logs: pairs of prompts marked "same intent" or "different intent". Then sweep thresholds:

// calibrate.ts interface Pair { a: string; b: string; sameIntent: boolean } function cosine(a: number[], b: number[]) { let dot = 0, na = 0, nb = 0; for (let i = 0; i < a.length; i++) { dot += a[i] * b[i]; na += a[i] ** 2; nb += b[i] ** 2; } return dot / (Math.sqrt(na) * Math.sqrt(nb)); } async function sweep(pairs: Pair[]) { const scored = []; for (const p of pairs) { const [ea, eb] = await Promise.all([embedText(p.a), embedText(p.b)]); scored.push({ sim: cosine(ea, eb), sameIntent: p.sameIntent }); } for (let t = 0.80; t <= 0.99; t += 0.01) { const hits = scored.filter((s) => s.sim >= t); const falsePositives = hits.filter((s) => !s.sameIntent).length; const truePositives = hits.length - falsePositives; console.log( `threshold=${t.toFixed(2)} hits=${hits.length} ` + `correct=${truePositives} WRONG=${falsePositives}`, ); } }

Choose the lowest threshold where WRONG is acceptably close to zero for your risk tolerance. For a customer-facing bot that quotes prices or policies, be very conservative. For a casual "explain this concept" endpoint, you can be looser.

Defense in depth beyond the threshold

  1. Segment by scope. Different products, plans, or la...