Prompt caching is one of the best features model providers shipped in the last two years. Anthropic, OpenAI, and Google all offer variants: pay less when the same prompt prefix shows up again [1][2][3]. For teams with huge system prompts or static tool schemas, the savings are real.
Then the category error creeps in. Someone on the infra channel says, "We have memory now. Look at the cache hit rate." That sentence should set off alarms. A cache is a discount on repetition. Memory is a decision about what to keep.
If you treat them as the same product, you will optimize the wrong curve. Your unit price falls. Your volume still explodes. The CFO sees a green dashboard and a red invoice.
What prompt caching actually does
Prompt caching stores a processed representation of a prompt prefix so later calls that share that prefix skip redundant work. Providers document minimum token thresholds, TTLs, and pricing tiers for cached versus uncached input [1][2]. The contract is narrow and honest: identical or overlapping prefixes get cheaper.
That is valuable. It is also not memory. Caching does not:
- Extract facts from yesterday's support ticket for today's chat
- Deduplicate the same policy pasted three ways across tools
- Isolate tenant A from tenant B
- Decide that a coding turn needs deeper context than a FAQ turn
- Stop an agent from replaying a growing transcript that never matches a stable prefix
If the bytes change every call, the cache cannot save you. Dynamic history is the default in production agents. So the cache helps the static shell and leaves the expensive middle untouched.
Why the bill still climbs
Imagine a team with a 12K-token system prompt that caches beautifully. Every user session still appends tool outputs, prior turns, and retrieved chunks. By turn 20 the dynamic portion dwarfs the cached prefix. You celebrate an 80% cache hit on the system prompt while paying full freight on a ballooning conversation body.
Research on long contexts makes that body dangerous as well as expensive. Models often under-attend to information placed in the middle of long prompts [4]. So you are not only paying more. You are paying to place facts where the model is least reliable.
Prompt caching reduces the cost of repeating the same prefix. It does not bound session growth. On workloads where full-context replay compounds, published measurements show prompt tokens climbing into the hundreds of thousands by late turns, while structured memory stays bounded [5]. Caching a bad architecture still leaves you with a bad architecture.
Memory decides. Caching discounts.
A memory layer asks different questions: What happened? What still matters? What is redundant? What is in scope for this tenant and this query? How many tokens does this call deserve?
Cortyxia sits at inference as that layer. It stores structured memory, retrieves what the query needs, and packs a bounded prompt. Prompt caching can still apply to stable envelopes. The difference is you are caching a lean, intentional prompt instead of discounting a landfill.
Teams that only buy caching optimize price per repeated token. Teams that buy memory optimize tokens per outcome. Only one of those curves bends the quarterly bill when product usage scales.
Key Takeaways
- Prompt caching discounts repeated prefixes. It does not create persistent, queryable memory.
- High cache hit rates can coexist with rising bills when dynamic history and tool output keep growing.
- Long prompts are not just expensive. Lost-in-the-middle effects can make oversized context less reliable.
- Memory systems decide what to store, retrieve, isolate, and inject. Caches only accelerate what you already send.
- Cortyxia reduces token volume at the source. Caching then discounts a smaller, stabler prompt.
Prompt Caching vs Memory — Frequently Asked Questions
The Bottom Line
Prompt caching is excellent infrastructure. It is not a memory strategy. If your plan is "cache the giant prompt," you are negotiating a discount on waste. Cortyxia removes the waste first: persistent memory, selective injection, bounded prompts. Then caching becomes a bonus on a system that already knows what to remember.