What Prompt Caching Actually Saves: The Token Math Behind LLM Gateways

Key takeaways

• Prompt caching reduces the cost of repeated input tokens (system prompts, few-shot examples, long context) by charging a fraction of the standard rate on cache hits.
• After caching kicks in, output tokens often make up 58-65% of total spend, shifting where you should optimize.
• Provider-agnostic caching in an LLM gateway lets you get caching benefits even when routing between providers, instead of losing the discount every time you switch models.
• Caching only helps when your prompts have a stable, reused prefix; highly dynamic prompts see little benefit.
• Model your before/after cost per request to see whether caching is worth the added routing complexity for your workload.

What is prompt caching and why does it matter for cost?

Prompt caching lets a provider reuse the processed representation of a prompt prefix (system instructions, long context, few-shot examples) across multiple requests, charging a steep discount, often 50-90% off standard input token rates, for the cached portion instead of full price every time. For workloads with a large, repeated prefix and a small variable suffix (a support bot with a long system prompt and short user messages, for example), this can cut input-token costs substantially.

Why do output tokens dominate after caching?

A recent breakdown of provider-agnostic caching in LLM gateways found that once caching is active, output tokens can account for roughly 58-65% of total spend, up from a much smaller share when input tokens were priced at full rate. This flips the usual cost-optimization instinct: teams often focus on trimming prompts and context length, but if caching already handles the input side, the bigger lever becomes managing output length, response verbosity, unnecessary elaboration, redundant formatting, since output tokens are typically priced several times higher than input tokens across most providers.

What does provider-agnostic caching add?

Caching implementations tied to a single provider only help as long as you stay on that provider. A provider-agnostic gateway applies caching logic at the routing layer, so switching models for cost, latency, or capability reasons doesn't mean losing the cache discount every time. This matters increasingly as teams route requests across multiple providers based on task type or price, a rigid single-provider cache defeats the purpose of that flexibility.

Is caching worth it for your workload?

Caching pays off when your prompts share a large, stable prefix across many requests, think a fixed system prompt, a shared knowledge base excerpt, or repeated few-shot examples, and less so for prompts that are mostly unique per request. Before adopting a caching gateway, model your current cost split between input and output tokens, and estimate the cached-hit rate you'd realistically achieve given how often your prefix repeats.

The takeaway

Prompt caching is a real cost lever, but only if your prompts have a reusable structure, and once it's working, output token management becomes the bigger optimization target. Calcaas lets you model cost per request with and without caching assumptions to see where the real savings are.