AI Atlas
Intermediate · ~4 min read #prompt-caching #cost #latency

Prompt Caching

Prefix caching · reading repeated input cheaply

Storing and reusing the KV state a model computed for a shared prefix (system prompt, tool definitions, a long document), so repeated input tokens are processed much more cheaply and faster.

SAME PREFIX → READ FROM CACHERequest 1system · tools · docsuser Awritten to cache · 1.25×Request 2system · tools · docsuser Bread from cache · 0.1×Request 314:03:27system · tools · docsuser Cmiss · recomputedcache breakpointprefix must match from the startINPUT TOKEN PRICE (NORMAL = 1×)normal1×write1.25×read0.1×TTL: 5 min (default)refreshed on every hitratios: Anthropic, 5-min TTL; varies by provider
Definition

As a model reads a prompt it computes attention key/value states for every token (the KV cache). When consecutive requests share the same beginning (same system prompt, same tool list, same 50-page contract), that work is redone every time. Prompt caching keeps the KV state of that shared prefix on the provider's side; if the next request starts with the same prefix, that part is read from cache instead of recomputed.

Four ideas show up at every provider:

  • Exact prefix match: the cache only covers the part that is identical from the very start of the prompt. One timestamp at the top, a reordered tool or a different image pushes everything after it out of the cache. Identical chunks in the middle don't help.
  • Cache breakpoint: the marker that says where the prefix ends (how far to write into the cache). Automatic at some providers, a field you set at others.
  • TTL (time to live): how long an unused entry survives. Usually minutes; some providers sell longer, hour-scale TTLs.
  • Write vs read pricing: tokens read from cache are much cheaper than normal input. Some providers charge more than normal input for tokens written to the cache, so caching pays off once a prefix is reused at least a few times.

Providers (per their official docs, November 2026):

  • OpenAI: on by default for supported models, no code changes. On GPT-5.6 and later the minimum prefix is 1,024 visible input tokens; cache reads cost 0.1× uncached input (0.05× on GPT-6.1 Sol and Terra). Cache writes cost 1.25× for the default 30m TTL and 2.5× for the 12h TTL in the Responses API. A breakpoint is placed automatically at the end of the latest eligible message, and you can mark your own with prompt_cache_breakpoint (up to four cache writes per request). Earlier models have no write charge; in_memory entries typically expire after 5–10 minutes idle (up to one hour), while 24h retention keeps them up to 24 hours.
  • Anthropic: enabled with cache_control: {"type": "ephemeral"}; at the top level of the request it is automatic, on content blocks it is an explicit breakpoint (up to 4). The default TTL is 5 minutes, refreshed for free on every hit; "ttl": "1h" gives 1 hour. Writes cost 1.25× for the 5-minute TTL and 2× for the 1-hour TTL; reads are 0.1× as standard (0.05× on Opus 5.5 and Sonnet 5.5, 0.025× on Fable 5.1 and Mythos 5.1). The minimum prefix ranges from 512 to 4,096 tokens by model; shorter prompts are processed uncached without an error. Each breakpoint looks back at most 20 blocks.
  • Google Gemini: implicit caching is on by default for all Gemini 2.5 and newer models; the minimum is 2,048 tokens on 2.5 models and 4,096 on Gemini 3.x and Gemini 4 Argon. There is no savings guarantee; hits are discounted automatically. Explicit caching (beta, v1beta) puts content into a cache object you manage and guarantees the saving; the TTL defaults to 1 hour with no minimum or maximum. You pay the discounted cached-token rate plus an hourly storage fee. Explicit caching works with generateContent on Gemini 3 and 2.5 models; it is not supported in the Interactions API or on Gemini 4 Argon and later.
  • DeepSeek: "Context Caching on Disk" is on by default for everyone. "Cache prefix units" are persisted at request and response boundaries, at detected common prefixes and at fixed intervals in long inputs; a later request must match a unit fully. It is best-effort with no hit guarantee, and unused cache is cleared within a few hours to a few days. No write charge: the pricing page lists deepseek-flash at $0.006 per 1M tokens on a cache hit vs $0.30 on a miss (peak; off-peak $0.003 vs $0.15).

These numbers change often; check the provider's pricing page and run your numbers in the cost calculator before deciding.

Analogy

Every day you hand a translator the same 40-page company glossary, then that day's one-page text. A smart translator doesn't reread the glossary each morning; they check yesterday's notes and go straight to the new page.

But the notes are tied to page order: put today's date on the glossary's first page and they decide it's a different glossary and reread everything. Nor are the notes kept forever; leave them untouched long enough and they get thrown out. That is prompt caching: as long as the fixed part stays first and unchanged, reading it is cheap.

Real-world example

A support bot sends a fixed 20,000-token prefix with every request: system prompt, tool definitions and product docs. The user message comes last, a few hundred tokens. The model is Claude Sonnet 4.6 (Anthropic pricing page: input $3, 5-minute cache write $3.75, cache read $0.30; all per 1M tokens).

For 100 requests arriving within five minutes, the prefix alone costs:

  • No cache: 100 × 20,000 = 2M tokens × $3 = $6.00
  • Cached: the first request writes (20,000 × $3.75/M = $0.075), the other 99 read (1.98M × $0.30/M = $0.594) → about $0.67

User messages and output are billed the same either way. If someone adds "Today: 8 November 2026 14:03:27" to the very top of the system prompt, the prefix changes on every request, every request pays the write rate again and the cache does more harm than good. The fix: move anything that changes to the end of the prefix, next to the user message. For more techniques see the token reduction guide.

Code examples
Anthropic: explicit breakpoint on a fixed system prompt (1-hour TTL) json
{
  "model": "claude-sonnet-4-6",
  "max_tokens": 1024,
  "system": [
    {
      "type": "text",
      "text": "<20,000 tokens of fixed instructions and product docs>",
      "cache_control": { "type": "ephemeral", "ttl": "1h" }
    }
  ],
  "messages": [
    { "role": "user", "content": "Where is my order?" }
  ]
}
Check hits in the response usage fields bash
# Anthropic: cache_creation_input_tokens (written), cache_read_input_tokens (read)
# OpenAI:    usage.input_tokens_details.cached_tokens and cache_write_tokens
# Gemini:    usage.total_cached_tokens
# DeepSeek:  usage.prompt_cache_hit_tokens and prompt_cache_miss_tokens
curl -s https://api.deepseek.com/chat/completions \
  -H "Authorization: Bearer $DEEPSEEK_API_KEY" -H "Content-Type: application/json" \
  -d @request.json | jq '.usage | {prompt_cache_hit_tokens, prompt_cache_miss_tokens}'
When to use
  • A long, fixed system prompt, tool list or document repeats on every request
  • Multi-turn chats and agent loops; the history is resent each turn with the same prefix
  • RAG or analysis flows that ask many questions about the same document in a row
  • When time to first token (TTFT) matters; a cached prefix is not recomputed
When not to use
  • The prefix is below the minimum (512 to 4,096 tokens depending on provider and model); it won't be cached anyway
  • Every request starts with a different document; with a write charge, cost goes up
  • Requests arrive less often than the TTL; you pay for a fresh write every time
Common pitfalls

Putting changing content first

A timestamp, user name, shuffled examples or session ID at the start of the prefix makes every request a miss. Fixed content first, variable content last.

Reordering tools between requests

Tool definitions are part of the prefix. Changing their order or schema from request to request breaks the cache; keep the list stable and append new tools at the end.

Forgetting the write charge

At Anthropic and on OpenAI's newer models, writing to the cache costs more than normal input. If a prefix is used once and discarded, caching raises cost; expect at least a few reads.

Trusting implicit caching without measuring

Gemini implicit caching and DeepSeek offer no savings guarantee, and at OpenAI hits depend on routing. Watch the cache counts in the usage fields and measure your actual hit rate.

Confusing it with response caching

Prompt caching only skips recomputing the input. The answer is still generated fresh and varies with settings like temperature; storing the same answer for the same question is a separate application-level cache.