Prompt caching explained: how it cuts the cost of AI agents

Prompt caching lets a model provider reuse the processed beginning of a prompt when a later request starts with exactly the same text. Cached input is billed at a fraction of the normal input rate, so it suits workloads that resend the same long context many times. AI coding agents are the clearest example: an agent resends its whole conversation on every turn, and without caching, most of a long session's bill is paying to process the same history again. Caching only works if the start of each request stays byte-for-byte identical, so it rewards agents that append to their context rather than rewrite it.

This page goes deeper on caching, which How much does AI coding cost? introduces.

How prompt caching works

Before a model generates anything, it processes every input token. For a long prompt that is real work, and providers bill for it as input tokens. If the next request begins with the same text, the provider can keep the result of processing that beginning (the prefix) and skip it next time.

Three properties follow from how this is done:

• It is a prefix match. The cache is used for the longest stretch at the start of the request that is identical to something cached. One changed character ends the match; everything after it is processed and billed normally.
• It expires. Caches live for a short time, typically minutes, and are refreshed when used. Some providers offer a longer lifetime at a higher write price.
• It is per model. A cache built for one model cannot be used by another, so switching models mid-session starts from nothing.

There is also a minimum length. Very short prompts are not cached at all.

How providers price it

The pricing has the same shape across providers, with different details:

• Cache reads are billed at a discount to normal input.
• Cache writes may cost more than normal input, the same, or nothing extra.
• Storage is charged by some providers for longer-lived caches.

Anthropic, for example, uses explicit cache markers in the request, matches in the order tools, system prompt, then messages, bills a write to the default five-minute cache at 1.25 times the base input price, and bills cache reads at a small fraction of the base input price, with the exact share depending on the model. OpenAI applies caching automatically to prompts above a minimum length, with no separate write charge. Google's Gemini API offers both automatic caching and explicitly created caches that are billed for storage time. Prices and rules change, so check the provider's documentation before you plan around them.

Why AI agents benefit so much

An agent session is a loop: send the conversation, get a response, run a tool, append the result, send the whole conversation again. Each call's input is the previous call's input plus a little more. That is exactly the pattern a prefix cache rewards.

An illustrative session, using token counts rather than prices so it holds for any model: 20 model calls, the first sending 20,000 tokens of context, each later call adding 4,000 new tokens.

• Without caching, total input across the session is 1,160,000 tokens, all billed at the full input rate.
• With caching, each call reads the previous context from cache and writes only the new 4,000 tokens. If a cache read costs a tenth of the input rate and a write costs 1.25 times it (illustrative rates; check your model's), the bill comes to roughly 226,000 tokens' worth of full-rate input: about a fifth of the uncached cost. A lower read rate saves more.

The longer the session, the larger the share of input that is cached history, and the bigger the saving. Output tokens are not affected by caching.

What breaks a cache

Because the match is on an exact prefix, small changes near the start of a request invalidate everything after them. Common causes:

• Volatile content early in the prompt. A timestamp, a request ID or a random value in the system prompt changes every call.
• Unstable ordering. Tools listed in a different order, or JSON serialised with keys in a different order, produce different bytes.
• Editing history. Rewriting or summarising earlier messages changes the prefix. Appending new messages does not.
• Changing settings mid-session. Switching models always starts a new cache, and on some providers changing certain request settings does too.
• Long pauses. If more time passes between calls than the cache lifetime, the next call pays to write the cache again.

The general rule: put stable content first and changing content last, and treat the conversation as append-only.

How to tell whether caching is working

Providers report cached tokens in the usage data of each response, typically as separate counts for tokens read from cache and tokens written to it. Three checks are worth doing:

1. Cache reads should be most of the input on every call after the first in a long session. If reads stay at zero, something early in the prompt is changing.
2. Writes should be small and steady, roughly the size of what each turn adds. Large writes on every call mean the cache is being rebuilt.
3. Cost per session should fall after you introduce caching or fix an invalidator. If it does not, the cache is not being hit.

Compare the cost of the same kind of session before and after a change, not individual requests.

Caching alongside other cost controls

Caching reduces the cost of repeated input. It does not limit how long a session runs, how many sessions people start or which model they choose. It works best with the other controls: shorter, well-defined tasks, the right model for each step, and a hard cap on spend. See How to set an AI coding budget for the last of those.

How AI CodeOps does it

AI CodeOps sessions that run on Claude models use prompt caching. The system prompt is cached as its own entry, and a marker at the end of the conversation caches everything up to its latest block, moving forward each turn, so an agent's tool loop resends its history as cache reads. Recorded cost counts cache writes and cache reads at their own rates rather than at the full input price.

You can see it working on the Usage page: in Usage History, open the input or output count of a model call to see its breakdown into cache reads, cache writes and reasoning tokens, where the model reports them. Each session's page shows its cost per agent step.

See Usage and spend for where these numbers appear, Models and tiers for which models sessions use, and estimate a session's cost with the free AI coding cost calculator.