Prompt caching

Before generating, a model must process your entire prompt. When the beginning of a request exactly matches the beginning of a recent one, same system prompt, same history, the network reuses that work instead of redoing it. You opt into nothing: the longest matching prefix is found automatically, time-to-first-token drops on the reused portion, and cached input tokens are billed at a discounted rate (live per-model rates on the dashboard's Models page). Responses are unaffected, a cache hit changes cost and latency, never output quality.

Seeing your hit rate

Cache hits show up in the response's usage as prompt_tokens_details.cached_tokens, the portion of prompt_tokens that was served from cache:

"usage": {
  "prompt_tokens": 5210,
  "completion_tokens": 318,
  "total_tokens": 5528,
  "prompt_tokens_details": {
    "cached_tokens": 4864
  }
}

Cached input tokens are also metered separately in your dashboard's Analytics tab, so you can watch the aggregate hit rate of a workload, not just one request.

Structuring prompts for cache hits

Matching is exact and prefix-only: one differing token invalidates everything after it. The rule that follows is simple, put what never changes first, and what always changes last:

# Cache-friendly: everything stable first, everything variable last.
messages = [
    {"role": "system", "content": STATIC_INSTRUCTIONS},   # identical every request
    *conversation_history,                                # grows append-only
    {"role": "user", "content": user_input},              # changes every request
]

# Cache-hostile: a timestamp up front breaks the prefix on every call.
messages = [
    {"role": "system", "content": f"Now: {datetime.now()}. " + STATIC_INSTRUCTIONS},
    ...
]

• Multi-turn chat caches naturally, each turn re-sends the history, which is a growing prefix of the last request. This is where the discount compounds most.
• Keep system prompts and tool definitions byte-stable. Timestamps, random IDs, or per-user greetings at the top of the prompt zero out the cache; if the model needs the current time, put it in the latest user message instead.
• Shared static context pays twice, a long document or codebase prefix queried repeatedly is processed once and reused across all the follow-up questions.

What caching does not promise

The cache is a best-effort accelerator, not a contract: entries are per-model, live for a limited time, and are evicted as capacity is needed. Never build correctness on a hit, treat cached_tokens: 0 as a normal, always-possible outcome that only costs you the regular input rate.