The problem it solves
Many AI applications send a big, fixed block of text with every request: a long system prompt, a set of tool definitions, reference documents, examples. That block does not change from call to call, but the model normally processes it fresh each time, and you are billed for all of it every time. If a fixed 10,000-word context goes out with a hundred different questions, you pay to process those same 10,000 words a hundred times.
That is pure waste, and at scale it dominates the bill. Prompt caching removes it. The stable part is processed once, then reused, so repeated context stops being charged at full rate on every call. For high-volume workloads with a large fixed prefix, the savings are substantial, and the response also comes back faster because the model skips redoing that work.
How it works
The mechanism is straightforward, and using it well comes down to one rule.
The stable prefix is remembered. The most expensive part of a request is the model’s first pass reading the input. When you mark a repeated prefix as cacheable, the provider stores the result of that first pass. On the next request that begins with the same prefix, it loads the saved work instead of recomputing it. The output is identical; only the redundant processing is skipped.
Order matters: static first, dynamic last. This is the whole game. Caching only works on a prefix that stays byte-for-byte the same, so anything unchanging, instructions, tool definitions, reference material, must come at the front, and anything that changes each time, the user’s specific question, at the very end. Put a changing value early and you break the cache for everything after it. Structuring the prompt so the stable part stays stable is the real work.
It pays off with repetition. There is usually a small extra cost to write something into the cache the first time, so caching wins when a prefix is reused enough to repay that. A few repeats and you are ahead; a prefix used once gains nothing. The exact discounts, timing, and controls differ by provider and change over time, so for current specifics the authoritative source is the provider’s documentation. What is stable is the principle: reuse the processing of repeated context to cut cost and latency.
A concrete example
Imagine a customer-support assistant with a long, fixed system prompt, detailed instructions, policies, and examples, that must accompany every question.
Without caching, every customer question re-sends and re-processes that entire block, and you pay for it each time. With caching, the fixed block is processed once and then reused across all the questions that follow, so you pay full price for it only at the start. The answers are the same; the repeated cost of the unchanging part largely disappears.
How it connects
Prompt caching is an optimization of how a repeated prompt and its context are processed, and its savings are measured in tokens, the unit of both cost and the thing being cached. It is one of the main levers for controlling cost and latency in a real system.
For the building roles it is practical craft. An AI/LLM Developer structures prompts so the cache actually hits, and an AI Solutions Architect designs systems whose fixed context is arranged to be cached efficiently.