Why AI Inference Costs Are Becoming the Real Battleground for SaaS Margins

Key takeaways

• Training is a one-time capex, inference is ongoing opex that scales with usage, meaning it shows up directly in your cost of goods sold (COGS), not a sunk R&D budget.
• Google reportedly processed 3.2 quadrillion tokens per month as of May 2026, roughly 300x its volume two years earlier, illustrating how fast inference demand is compounding industry-wide.
• "Tokens per watt" is emerging as the core efficiency metric for inference hardware, and improvements there eventually flow into what LLM providers charge per token, though not always immediately.
• Batching (serving many requests together) cuts cost per token but adds latency, a trade-off that shows up in real API pricing tiers (real-time vs. batch endpoints) and is worth modeling explicitly.
• Pricing your product by seat instead of usage can hide the fact that your actual inference COGS is moving, sometimes for months, before anyone notices the margin drift.

Why does inference cost more attention than training?

Training is the cost everyone talks about because it is large, dramatic, and finite: a company builds a model, spends a known and typically one-time sum, and moves on. It behaves like capex.

Inference is different. Every time a user sends a prompt, opens a chat, or triggers an autonomous agent step, a model has to read its weights out of memory, run the math, and produce a token, then do it again for the next token. That happens every single time someone uses your product. It is an operating cost that scales linearly (or faster) with usage, which is exactly the kind of cost that erodes margin quietly if you are not tracking it per unit.

For SaaS teams building on top of LLM APIs, this is the number that actually determines whether an AI feature is profitable at scale, not the sticker price of the model.

What is "tokens per watt" and why should founders care?

As inference volume has scaled (Google alone reportedly moved from roughly 1/300th of its May 2026 token volume just two years prior), the industry has started optimizing hardware specifically for the shape of inference work: streaming huge weight files from memory, running simple math on each byte, over and over. General-purpose GPUs are flexible but were not built only for that job.

New inference-specific hardware is being built around "tokens per watt," meaning how many tokens a chip can generate per unit of power, since compute-intensive AI infrastructure is now often constrained by power availability, not just dollars.

For a founder pricing an AI product, this matters indirectly: as tokens-per-watt improves, the underlying cost of serving a token should trend down over time. But list prices from providers do not always fall in lockstep, especially while providers are still recovering capex on frontier models. That gap, between what inference could cost given the hardware trend and what you are actually being billed, is worth watching rather than assuming away.

How does the batching trade-off affect your pricing model?

One practical lever inference providers use today is batching: grouping many requests together so the expensive read of model weights gets amortized across more tokens. It lowers cost per token but adds latency, since a request has to wait for a batch to fill.

This is not an abstract systems detail. It already shows up in your API bill: several providers offer discounted batch endpoints (commonly around half the price of real-time calls) specifically because batching is cheaper to serve. If part of your product does not need sub-second responses, for example nightly summarization or asynchronous agent tasks, routing that traffic through a batch tier is a straightforward margin lever, and one that is easy to model before you build it.

The original observation: seat-based pricing can hide moving COGS

Here is the piece that gets missed in most inference-cost discussions: if you charge per seat rather than per usage, your pricing is disconnected from your actual cost driver. Inference cost per query can be falling (as specialized hardware improves tokens per watt) or rising (as your users adopt heavier agentic workflows with longer context windows), and your seat price will not reflect either move until someone runs the unit economics.

The fix is not necessarily to switch to usage-based pricing. It is to know, per plan tier, roughly what your blended cost per active user actually is today, and to re-check that assumption on a cadence, not just at launch. That is the kind of scenario you can model in Calcaas: simulate a user tier's expected token volume against current provider pricing, and see the margin before a hardware or pricing shift catches you off guard.

Takeaway

Training got the headlines, but inference is the recurring bill, and as the industry pushes for more tokens per watt, the founders who track their actual per-query cost, not just their sticker-price assumptions, will be the ones who protect margin as the underlying economics shift.