The Complete Guide to Prompt Caching: How It Works, Cost Model, and Implementation

What problem does prompt caching solve

In most LLM calls, a substantial portion of the input is the same every time: system prompts, few-shot examples, retrieved fixed documents, and tool definitions (tool schema). Under normal billing, you pay for these repeated input tokens every time, and the attention is recomputed every time as well.

The idea behind prompt caching is straightforward: the server stores the computed result of this prefix, and on the next request, if the prefix is exactly the same, it reuses it directly and only recomputes the newly added part. Anthropic positions it in its [prompt caching documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) as a way to reduce latency and cost; on the OpenAI side, the corresponding mechanisms are automatic prompt caching and batch discounts.

Which scenarios actually save money

Prompt caching is not a switch you flip to save money. It has a clear break-even boundary: number of times the same prefix is reused × prefix length > the extra overhead of writing the cache.

• Long system prompt + multi-turn conversation: system prompts of several thousand tokens with many conversation turns almost always pay off.
• RAG + fixed document set: put unchanging policy documents/knowledge bases in the prefix and let only the user question vary.
• Code agents / tool calls: tool schemas and repository structure descriptions are usually repeated in every call.
• One-off calls with short prefixes: the cost of writing the cache may exceed the savings; do not force caching.

Five common mistakes that invalidate the cache

• Mixing timestamps or random IDs into the prefix: even a single Date.now() at a fixed position changes the entire prefix.
• Dynamically assembling the system prompt: different users get different prefixes, diluting the cache hit rate to near zero.
• Summarizing and compressing historical messages: rewriting once per turn makes the prefix unstable.
• Putting user input first: this is the most common structural mistake; variables should be as late as possible.
• Looking only at the bill and not the hit rate: without a hit rate metric, cache benefits cannot be attributed at all.

Reusing the same caching strategy across multiple models

If your product uses both Claude and GPT, cache semantics are not fully consistent across vendors (who declares cache breakpoints, how long the TTL is, how billing works). Putting this adaptation behind a unified gateway has the benefit that business code maintains only one prompt structure and does not need to be rewritten when switching models.

Admesh's gateway maintains an OpenAI-compatible protocol downstream, so the same prompt structure can switch between vendors by changing only the model field; cache hit billing is reflected uniformly in the usage details of the bill, so you do not need to integrate each vendor's billing semantics separately.

View billing units and cached token documentation →

Implementation checklist

1. Split the prompt into three parts: 'stable prefix / semi-stable part / user input', with the stable part first.
2. Ensure the stable prefix contains no timestamps, random numbers, or user IDs.
3. Track cache read/write tokens separately in the bill and convert them into a real hit rate.
4. Use a fixed set of evaluation requests for regular regression to confirm that prompt changes have not broken the hit rate.
5. Set a lower bound for cache benefit (for example, roll back the prompt structure if it falls below 5%) to avoid optimizing for optimization's sake.