Prompt caching for LLM APIs: 6 reasons your cache never hits

Originally published on Extenify.

Prompt caching can make the repeated part of an LLM prompt up to 90% cheaper on OpenAI, Claude and Gemini, but a lot of backends turn it on and then never get a single cache hit. The bill looks normal, cached_tokens sits at 0, and nobody notices. Here is how the cache decides what to reuse, when it saves money (and when it costs more), and the six usual reasons it stays cold.

How the cache decides what to reuse

Every provider caches a prefix: the tokens at the very start of the request, which must match byte for byte from the first token. Once something differs, everything after it is a miss.

• Claude reads the request in this order: tools, then system, then messages. OpenAI and Gemini also match from the start of the request.
• Only the prompt is cached, not the answer. The model still generates a fresh response; the reused tokens are just cheaper and the first token arrives sooner.
• Entries expire after minutes or hours, and an expired entry has to be written (and on some models paid for) again.

So the rule for any prompt template is: stable content first, changing content last. A cache-friendly request looks like this:

1. Tool definitions, in a fixed order and serialized the same way every time
2. System instructions, with no timestamps, user names or request IDs
3. Shared reference material: schemas, style guides, few-shot examples
4. Per-conversation context, such as the page a user is reading
5. The user's actual question

Parts 1 to 3 are shared by all users; part 4 is reusable within one conversation.

The numbers, side by side (October 2026)

These figures change often, so check the provider's own docs for your exact model before you budget around them.

The break-even math

Now that OpenAI's newer models and Claude both charge to write the cache, caching a prefix that is never reused costs you more than not caching it. With a 1.25x write and a 0.1x read, the cost relative to plain input looks like this:

A 5-minute cache pays off as soon as a prefix is used twice; a 1-hour cache needs three uses. Gemini's explicit caches charge storage per hour instead, so they only make sense for a large document that gets read several times an hour.

6 reasons your cache never hits

1. The repeated part is too short. The total prompt can be well over 1,024 tokens, but only the identical prefix counts. A 400-token fixed system prompt followed by a long dynamic message will never cache.
2. Something near the top changes. A "current date and time" line in the system prompt is the classic case. So are JSON objects serialized with keys in a different order, or tool lists built from a Set.
3. Tools or settings changed. On Claude, editing tool definitions invalidates the whole cache, and changing tool_choice, images or thinking settings invalidates the messages. Keep the tool list stable and narrow it with tool_choice instead of removing tools.
4. The entry expired. Claude's 5-minute window counts from the start of the request, and streaming time counts against it. A long answer leaves little time for the follow-up.
5. Requests go to different caches. Caches are isolated per organization (and per workspace on Claude), and OpenAI's are per machine. Your staging key and production key never share one.
6. The model's minimum went up. Moving to Gemini 3.8 Flash raises the minimum to 4,096 tokens, and some older Claude models need 4,096 too.

Measure the hit rate on every response

Each provider reports cache usage in a different field, so normalize them once and log the result with the feature name and prompt version:

// Normalize cache usage across providers: { input, read, write } export function cacheStats(provider, res) { switch (provider) { case 'anthropic': { const { input_tokens, cache_read_input_tokens: r = 0, cache_creation_input_tokens: w = 0 } = res.usage; return { input: input_tokens + r + w, read: r, write: w }; // input_tokens excludes cached parts } case 'openai': { const d = res.usage.input_tokens_details || {}; return { input: res.usage.input_tokens, read: d.cached_tokens || 0, write: d.cache_write_tokens || 0 }; } default: { // gemini: promptTokenCount already includes cached tokens const m = res.usageMetadata; return { input: m.promptTokenCount, read: m.cachedContentTokenCount || 0, write: 0 }; } } } 

Plot read / input per feature per day. The exact level matters less than sudden changes: if it falls to near zero right after a deploy, someone put something dynamic at the top of the prompt, and an alert catches it before the invoice does.

Key takeaways

• Prompt caching matches an exact prefix, so put stable content first and keep it byte-identical.
• With write charges on newer OpenAI models and on Claude, a prefix used only once costs more than no caching; two uses on a 5-minute cache already pay off.
• Minimum prefix sizes run from 512 to 4,096 tokens; shorter prompts are simply not cached, with no error.
• Version your prompt templates and log cached vs. written tokens from every response so a broken cache shows up as an alert.

The original article has the full provider comparison and ready-to-use Node.js fetch examples for Claude, OpenAI and Gemini explicit caches in the complete prompt caching guide for OpenAI, Claude and Gemini. If you ship an AI-powered browser extension, Extenify tracks your extension's reviews, ratings and weekly users across Chrome, Edge and Firefox, so you see when slow or wrong answers start to hurt.

What hit rate are you actually seeing in production, and what was the sneakiest thing that broke your prompt cache?