Anthropic Prompt Caching: How to Stop Paying to Re-Read Your Own System Prompt

Your input tokens are probably the same tokens, over and over, every single request — and you’re paying full price to have the model re-read them each time. That’s the whole problem prompt caching solves, and it’s the highest-leverage cost fix available to anyone shipping a chat, agent, or RAG feature on Claude or GPT models today.

TL;DR: Anthropic prompt caching lets you mark a prefix of your prompt — system instructions, tool definitions, few-shot examples, a big RAG context block — as cacheable. The provider stores the pre-computed attention state for that prefix, so the next request that shares it skips re-computation and pays a fraction of the normal input price for those tokens. OpenAI does something structurally similar but automatically, with different economics and a different failure mode. Both require engineering discipline, not just a config flag, to actually pay off.

Direct Answer: What Anthropic Prompt Caching Is

Anthropic prompt caching is a mechanism where you designate parts of your prompt — via explicit “cache breakpoints” in the request — as reusable across calls. As widely reported by developers and pricing write-ups, when a subsequent request has a prefix that matches a previously cached one, the matching portion is served from cache instead of being recomputed, and billed at a discount relative to standard input tokens — though this should be checked against current published rates before you rely on it. It’s opt-in and explicit: you choose where the cache boundary sits.

That’s the core distinction from OpenAI, whose caching is automatic and requires no special parameters in the request. Both aim at the same target — repeated prefixes — but the operational model, and the discount you get, differ enough that “just turn it on” isn’t the whole story for either provider.

How Prompt Caching Works

Every LLM request that includes a prompt gets processed through an attention mechanism that computes key-value (KV) tensors for each token — essentially, an internal representation the model uses to attend back over earlier tokens as it generates output. Computing those tensors is a meaningful share of the cost and latency of a request, especially for long prompts. Prompt caching stores those pre-computed KV tensors for a prompt prefix on the provider’s side, so a later request sharing that exact prefix can reuse the stored tensors instead of recomputing them from scratch — only the new, dynamic suffix needs fresh computation.

The critical word there is exact. Cache matching is byte-for-byte: change one token anywhere inside the cached prefix — an injected timestamp, a request ID, a session counter, a reordered JSON key — and the cache misses entirely, and you pay full price to recompute the whole prefix. This is the single most common way teams silently lose the discount: something in what they assumed was “static” content is actually drifting request to request.

Because the mechanism operates on a shared prefix, structure matters more than content. A cache hit only covers the portion of the prompt that’s identical from the start of the request through to the point where it diverges from a previously cached version. Anything after the first point of difference is uncached, no matter how much of the rest is technically repeated content elsewhere in the prompt.

How to Enable Anthropic Prompt Caching

Anthropic’s caching model is explicit: you mark specific points in your prompt — after your system instructions, after a block of tool definitions, after a large embedded document — as cache breakpoints, telling the API “everything up to here is cacheable and should be checked against the cache.” This gives you control over exactly what gets cached and where the boundary sits, which matters when your prompt mixes long static context with short dynamic content.

As commonly documented, cached entries have a time-to-live (TTL). The practical implication for engineering is straightforward: if requests against a given cached prefix arrive frequently enough to fall within the typical TTL window, you keep hitting cache; if traffic is bursty or sparse, entries expire and you’re back to full-price writes. This is why cache-aware system design — batching similar requests, keeping session-affinity where possible — pays off on top of just enabling the feature, though the exact TTL window should be checked against current published rates.

As typically described in pricing write-ups, you can also extend the cache lifetime for a higher write cost, which is a real tradeoff: a longer TTL costs more upfront to write but protects you against low-frequency traffic patterns that would otherwise miss cache repeatedly. Whether that trade is worth it depends on your request cadence — for a system prompt hit thousands of times an hour, the default short TTL is probably fine; for a nightly batch job with hours between calls, the extended TTL earns its premium — though the exact costs and TTL windows are worth checking against current published rates directly.

What Anthropic Prompt Caching Costs

As commonly documented across pricing analyses, the economics work on three token categories: standard input tokens (full price), cache write tokens (writing a new prefix into the cache), and cache read tokens (hitting an existing cache entry). As typically described, cache writes cost more than a standard input token because you’re paying to populate the cache in addition to processing the tokens, while cache reads cost dramatically less because you’re skipping the recomputation almost entirely. The figure widely cited across pricing analyses is that cached input tokens run at roughly a 90% discount versus standard input pricing — the exact multiple depends on the model and should be checked against current published rates before you build a cost model around it, since list prices move.

That 90% figure is the reason prompt caching is frequently cited as the single highest-ROI lever for cutting LLM input costs — ahead of switching models, compressing context, or restructuring outputs — precisely because it requires no quality tradeoff at all when implemented correctly. It’s pure architecture, not a model downgrade.

OpenAI’s Automatic Caching, Contrasted

OpenAI takes the opposite posture on opt-in: there’s no explicit breakpoint parameter to set. Caching kicks in automatically once a prompt prefix is long enough and repeats, routing based on a hash of the leading portion of the prompt. The practical consequence is that you get less manual control over what’s cached, but also nothing to misconfigure — the system decides for you based on prefix length and repetition.

The two models trade off differently on discount depth versus TTL flexibility and control granularity. Anthropic’s explicit-breakpoint approach gives you precision over where the cache boundary sits, at the cost of needing to actively structure prompts and manage breakpoints. OpenAI’s automatic approach removes that management burden, but you have less say over exactly which prefix gets treated as cacheable. Based on each provider’s publicly documented pricing, the two providers land on different discount depths for cache reads, with Anthropic’s cited cache-read discount deeper than OpenAI’s — though it’s worth checking both providers’ current published rates directly, since these figures move. Given that gap, if you’re deciding where to invest engineering effort first, Anthropic’s explicit model rewards careful prompt structuring more than OpenAI’s does, because you have more levers to pull and more ways to get it wrong.

Both providers share one structural constraint worth internalizing: there’s a minimum prefix length before caching applies at all. A prompt that’s mostly short dynamic content with only a small static header won’t benefit much from either provider’s caching — the fixed overhead of caching only pays off once the static prefix is substantial.

How to Maximize Cache Hits

The rule that matters more than any API parameter: structure every prompt static-before-dynamic. Put everything that doesn’t change — system instructions, persona definitions, tool schemas, few-shot examples, formatting rules — at the very front of the prompt, and put everything that does change — the user’s latest message, a timestamp, retrieved documents that vary per query — at the end. Since the cache matches on prefix, the longer your identical leading block, the more tokens get the discount, and the smaller the “wasted” recompute tail becomes on every call.

Practical patterns that follow from this:

• Freeze your system prompt. Don’t template in per-request values like the current date or a user’s name at the top of the prompt — push those into the dynamic tail instead.
• Keep tool definitions stable and grouped early. If your tool schema list changes per request (even reordering), you lose the cache on that entire block.
• Batch RAG context consistently. If retrieved documents are appended after static instructions but before the user query, you preserve the cacheable prefix even as the retrieved content varies request to request.
• Watch your conversation history growth. In multi-turn chat and agent loops, the LLM API is stateless — the full conversation history gets re-sent on every turn. Caching flattens the cost curve of that growth, but if content injected early in the conversation (like a running summary that updates each turn) changes upstream of later turns, you can invalidate the very prefix you were relying on to stay cached.

Cache Invalidation Cascades

Invalidation doesn’t happen in isolation — it cascades, and understanding the cascade order is what separates “we enabled caching” from “we actually see the savings.” Changing your tool definitions invalidates the entire cache, because tool schemas typically sit early in the prompt and everything downstream of a changed byte is recomputed. Changing the system prompt invalidates both the system-level cache and any message-level cache built on top of it. Changing only the trailing user message, by contrast, can leave your system prompt and tool-definition caches intact, since nothing upstream of that final turn changed.

This is why the order you structure prompts in isn’t just a nice-to-have — it determines the blast radius of any given change. A one-line tweak to a system prompt that ships several times a day can quietly reset your cache hit rate to zero until traffic re-warms it.

Cache-Unaware Routing: The Silent Killer

The failure mode that catches teams off guard isn’t usually the prompt — it’s the infrastructure around it. A load balancer or API gateway that routes requests round-robin across model workers, or a system that swaps between model versions for A/B testing or fallback, can send what should be an identical, cacheable prefix to a different backend on every call. If the underlying worker or model instance changes, the cache doesn’t carry over, forcing a full re-prefill regardless of how carefully you structured the prompt. The same applies if your own retry or fallback logic silently switches models mid-session — a fallback from a primary model to a backup on error looks harmless functionally, but it resets the cache for that conversation. If you’re running any kind of gateway, proxy, or multi-model routing layer in front of Anthropic or OpenAI, cache-awareness needs to be a routing constraint, not an afterthought.

Worked Example: Estimating Your Own Savings

Consider a hypothetical support bot with a 2,000-token system prompt — persona, guardrails, formatting instructions — sent as the prefix on every request, at a volume of 10,000 requests a day. Without caching, all 2,000 tokens are billed at full input price on every single call. With caching, as commonly documented, only the first request (or the first after a TTL expiry) pays the write cost; the rest read from cache at the discount described above. At meaningful daily volume, the compounding effect of the commonly-documented cache-read discount on a fixed 2,000-token block repeated 10,000 times a day is the theoretical difference between a system-prompt cost that scales linearly with traffic and one that’s nearly flat — though real end-to-end savings depend on how much of your traffic actually shares a stable prefix, and are worth validating against your own usage and both providers’ current published rates directly rather than assumed from the model alone.

Now consider a busier agentic scenario: 1,000 requests per hour, each carrying a 5,000-token system prompt plus tool definitions and a short 300-token user query. Without caching, every request pays full input price on the entire 5,300 tokens. With a high cache hit rate — say 90% of requests landing on a warm cache — the bulk of that 5,000-token static block is billed at the discounted cache-read rate instead of full price, and only the 300-token dynamic tail (plus the occasional cache-write refresh) is billed normally. The gap between the no-caching cost and the cached cost at that hit rate is large enough that it changes whether the agent is economically viable at scale.

One honest caveat: theoretical cache-read discounts and real end-to-end savings aren’t the same number. If your prefix changes frequently enough that cache writes happen often relative to cache reads, or if a chunk of your “static” content is quietly drifting per request, your realized savings will land below the headline discount — sometimes meaningfully below it. Model your savings on your actual hit rate, not the advertised ceiling.

Prompt Caching vs. Exact-Match and Semantic Caching

It’s worth being precise about what prompt caching is not. Provider prompt caching operates at the token level inside a single request — it reduces the cost of recomputing a prefix the model has seen before. It says nothing about whether the response to a semantically identical question gets reused.

Exact-match response caching sits a layer above: if you’ve literally seen this exact input before (same hash, same request), you serve the stored output without calling the model at all. Semantic caching goes further — it tries to match requests that mean the same thing even if worded differently, typically via embedding similarity, and serves a cached response if the similarity clears some threshold. Both operate at the application layer, independent of the provider, and both carry their own correctness risk: exact-match caching is safe but narrow, while semantic caching risks serving a stale or subtly wrong answer if the similarity match is too permissive.

The practical ordering: implement provider prompt caching first, since it’s essentially free correctness-wise — you’re not changing what gets served, only how cheaply the model re-reads its own context. Exact-match and semantic caching are separate, optional layers on top, worth adding once you’ve captured the low-risk win that prompt caching represents.

The Bottom Line

Prompt caching isn’t a checkbox — it’s a prompt-structuring discipline. The technical lever (explicit breakpoints on Anthropic, automatic routing on OpenAI) is the easy part. The hard part is keeping your static prefix genuinely static across thousands of requests, understanding which changes cascade into full invalidation, and making sure nothing in your routing or infrastructure layer is quietly defeating the cache before it ever reaches the model. Get that discipline right, and it’s the rare cost optimization that costs you engineering time once and pays out on every request after.

FAQ

Does prompt caching change the model’s output quality? No — caching only affects how the prefix is computed and billed, not what the model generates. It’s a cost and latency optimization, not a quality tradeoff.

Do I need a minimum prompt length before caching helps? Yes, in practice — both major providers require a prefix of meaningful length before caching engages at all, so very short prompts won’t see much benefit.

Can I use prompt caching and semantic caching together? Yes — they operate at different layers and solve different problems, so there’s no conflict in running both, provider prompt caching for token-level reuse and semantic caching as an application-layer response shortcut.

What’s the single most common reason teams don’t see the expected savings? Injected variable content inside what they believe is a static prefix — timestamps, IDs, or reordered fields — breaking the byte-for-byte match, or a cache-unaware router sending requests to different backend workers.