Prompt Caching Mechanics

Prompt caching reuses a stable prefix server-side, but only the content before the first change stays cacheable.

Context & Reliability·Lesson 2 of 6·7 min

Prompt caching lets you mark a portion of your request as a reusable prefix so that, on subsequent requests, the model provider can skip reprocessing that prefix from scratch and instead read it from a server-side cache. This cuts both latency and cost on the cached portion, which matters enormously for agentic systems that send large, mostly-static context (system prompts, tool definitions, long documents) on every single turn of a conversation.

The prefix rule

The mechanic that matters most for the exam is this: caching works on a prefix basis. Everything up to and including your cache breakpoint is a candidate for reuse, but only if the bytes in that region are byte-for-byte identical to a previous request. The moment content changes anywhere before your cache boundary, the cache is invalidated for everything from that change point onward, even if 95% of the prefix is unchanged. This is why the ordering of content within a request is not a stylistic choice; it is the single largest lever you have over your cache hit rate.

The practical consequence is that you must design your requests so that stable content comes first and variable content comes last. A system prompt, tool definitions, and any large static reference documents belong at the front of the request, marked with a cache breakpoint. Conversation history that grows turn by turn should be appended after that breakpoint. If you instead prepend new information, inject a timestamp into the system prompt, or reorder tool definitions between requests, you silently break caching for the entire prefix that follows, and you will not necessarily get an error, just a much higher bill and higher latency than expected.

Where this goes wrong in agent loops

A common mistake is including something trivially dynamic inside the cached region: a "current date" field, a random request ID, or a live token count injected into the system prompt on every call. Any one of these changes the prefix on every request and defeats caching entirely, even though the bulk of the system prompt is identical every time. The fix is to move genuinely dynamic content out of the cached prefix and into the part of the request that comes after the cache breakpoint, or to accept that highly dynamic fields cannot be cached and isolate their cost.

Another failure mode is in multi-turn agent conversations where each turn's tool results get inserted in the middle of the history rather than appended at the end. If your framework reorders or summarizes earlier turns before the current one is sent, you invalidate the cache for the entire conversation on every single turn, which is exactly the scenario prompt caching is meant to optimize away.

Exam trap A question presents a system prompt that begins with "Today's date is [dynamic date], you are a helpful assistant..." followed by a long, otherwise-static block of instructions and tool definitions, and asks why cache hit rates are low despite the prompt "mostly staying the same." The trap answer blames the model or the tool definitions. The correct diagnosis is that the dynamic date sits before the cache breakpoint (or at the very start of the prefix), invalidating the cache on every request regardless of how much of the rest is identical. The fix is moving dynamic fields after the cached block, not shortening the prompt.

Scenario: A support agent sends a request each turn containing: a 2,000-token system prompt, 1,500 tokens of tool definitions, a growing conversation history, and the current user message. The team notices cache hit rates are near zero even though the system prompt and tools never change. Investigation shows the conversation history is being summarized and reinserted near the top of the message list on every turn, ahead of the cache breakpoint, to "keep things tidy." The reasoning error: summarization is useful, but placing its output before the cache boundary — or restructuring anything before that boundary — invalidates the cache for the system prompt and tools that follow, even though those parts are unchanged. The fix is to place the cache breakpoint after the system prompt and tool definitions, and to only ever append new content afterward, keeping any summarization strictly downstream of the cached region.

Try it

Send the same multi-turn conversation twice through the API with a cache breakpoint set after your system prompt and tools: once appending each new turn normally, and once where you insert a dynamic field (like a live timestamp) into the system prompt before the second call. Compare the cache_read_input_tokens and cache_creation_input_tokens fields in the response usage between the two runs to see the invalidation directly.