Latency Now Has a Price Tag: What Faster Inference Is Actually Worth

Key takeaways

• Claude's Fast Mode runs $10 per million input tokens and $50 per million output, precisely 2x the standard $5/$25 rate. Speed is now an explicitly priced product.
• French startup Kog is betting the opposite way: extract more throughput from standard datacenter GPUs enterprises already own, rather than buying faster silicon or paying a premium tier.
• Its May 2026 tech preview hit 3,000 tokens per second on a single request using AMD MI300X and Nvidia H200 hardware, but with a purpose-built 2B model, Laneformer 2B, now open sourced.
• That gap between a 2B demo and a production-scale LLM is the entire investment case, and the company expects a 10x result on a major model around September.
• The founder takeaway is not about Kog. It is that you can now compute what a latency improvement is worth instead of guessing.

Why does a Fast Mode premium matter more than a benchmark?

Because it turns latency from an engineering preference into a line item with a number attached.

When a frontier lab charges exactly 2x for the same model served faster, it has published a market price for speed. That number is useful whether or not you ever buy the fast tier. It tells you the most a rational buyer in that market will pay for a latency improvement, which means it also tells you the most a speed optimization is worth to you.

Run it as a simple test. If a change to your serving stack gives you 3x the throughput on hardware you already rent, and the market price of a 2x speed improvement is a 100% premium, that optimization is worth more than the engineering time almost every time. If it gives you 10% and costs a quarter of engineering, it is not.

Most teams never do this calculation because latency sits in the engineering column and pricing sits in the finance column. It belongs in both.

What is Kog actually claiming?

Kog is a French startup with a team of eleven, betting that conventional GPUs still have far more inference performance available than anyone extracts. Its tech preview aimed to prove that, in the company's words, extremely fast single-request decoding is possible on the standard datacenter GPUs enterprises already own, and it used AMD MI300X and Nvidia H200 cards for the demo.

The headline number is 3,000 tokens per second per request. The caveat is equally important: that was a purpose-built model of roughly 2 billion parameters, the now open-sourced Laneformer 2B, not a frontier LLM. CEO Gaël Delalleau's stated promise is 30x faster LLM inference, and he expects a first major model at 10x speed around September.

The demand signal is real regardless. Kog reported 200 tangible business leads off the preview, mostly from teams whose professional workflows stall on inference latency. Veteran Claude Code users know the shape of that problem: sometimes you wait hours for a result.

Is this a hardware problem or a software problem?

This is where the contrarian view earns its keep. The dominant narrative in 2026 is that conventional GPUs are poorly suited to agentic decoding, which is why purpose-built inference chips attracted so much capital, including Cerebras's May 2026 IPO after a $5.5 billion raise.

Delalleau calls that a misconception, arguing that newer GPUs have increasing memory bandwidth that simply is not being unlocked. Kog is not alone in that view. ZML, also French, released hardware-agnostic software that bypasses CUDA to speed inference across competing chips.

If they are even partly right, the practical implication for founders is uncomfortable in a useful way: some meaningful share of what looks like a hardware budget problem is an unexploited software problem, and you are currently paying a premium to route around it.

The honest counterweight is that Kog has not yet demonstrated this on a large model, the approach is hands-on enough that each new GPU takes weeks or months of engineering, and eleven people can only cover so many chips.

How should you price latency in your own product?

Three steps, in order.

First, measure what latency costs you today, in both directions: the premium you pay a provider for faster serving, and the revenue you lose to abandoned sessions or slow agent loops.

Second, set a ceiling using the market price. A 2x premium for a meaningful speed improvement is the reference point currently on the table.

Third, decide whether latency is a cost or a product. If your users would pay for speed, a fast tier is a margin opportunity, not just an expense. Providers have already proved the willingness to pay exists.

One line to keep: the moment someone publishes a price for speed, speed becomes something you can model rather than argue about.

You can put your own latency premium into a per-request cost at calcaas.com.