Your Load Balancer Is Quietly Inflating Your LLM Bill

Key takeaways

• Prefix caching lets a server skip recomputing a prompt it has already seen, and providers often bill those cached tokens at a steep discount.
• A standard load balancer routes by chance, so repeat requests rarely land on the server holding the cache. The cache misses, and you pay full price.
• Prefix-aware (KV-cache-aware) routing sends requests with the same prefix to the same server, recovering the discount and cutting latency.
• The cache discount you penciled into your margin model is optimistic unless your routing is cache-aware.
• Model both a naive and a cache-aware hit rate before you trust the savings.

What is prefix caching, in plain terms?

When a model reads your prompt, it builds an internal cache (the KV cache) for those tokens. If the next request starts with the same prefix, say the same long system prompt, the server can reuse that work instead of redoing it. Less compute means lower cost and faster responses, and many providers bill those reused input tokens at a large discount. For apps that send the same system prompt on every call, prefix caching is one of the easiest wins on the menu.

Why do standard load balancers break it?

A standard load balancer spreads requests evenly or randomly across servers for resilience and balance. That is exactly wrong for caching. The cache lives on whichever server handled the first request. When your repeat request lands on a different server by chance, that server has no cache, so it recomputes from scratch and you pay full price. The cruel twist: the more servers you add to scale, the lower your odds of a cache hit. Your infrastructure ends up fighting your cost optimization.

How big is the hidden cost?

It is the gap between the cache discount you assumed and the hit rate you actually get. Say you modeled an 80% cache hit and your routing delivers 20%. Most of the discount you priced into your margins never shows up. That number is illustrative, but the trap is real: the savings look great on paper and quietly evaporate in production, and nothing in your dashboard screams about it.

How do you fix it? Prefix-aware routing

The fix is to route by content, not by chance. Prefix-aware, or KV-cache-aware, routing inspects the start of each request and sends matching prefixes to the same server, so the cache is there when the request arrives. You keep the resilience of running multiple servers and stop throwing away the cache on every other call. Hit rates climb, latency drops, and the discount you modeled finally lands in the bill.

What this means for your pricing model

Here is the contrarian part. Most pricing spreadsheets treat the cached-token discount as a flat percentage, a clean line in the margin math. It is not clean. Your real discount depends on infrastructure you may not have tuned, and it can swing with a config change. Before you bank those savings in a plan or a price cut, model two scenarios: a naive hit rate and a cache-aware one. The gap between them is your margin's exposure to a routing decision most teams never audit.

The takeaway: prefix caching only pays off if your routing respects it, so model the realistic hit rate before you price against it. You can run both the naive and cache-aware scenarios on token costs and margins in Calcaas in a few minutes.