Cheaper Tokens Do Not Equal Better Margins: What GPT-5.6's Efficiency Push Means for Your AI Unit Economics

Key takeaways

• OpenAI reports a 20% cut in end-to-end serving costs from kernel optimizations, and a 15%+ improvement in token-generation efficiency from a better speculative-decoding draft model.
• Tiering, not haggling, is the biggest price lever available to you: Luna is priced 80% below Sol, and Terra is described as matching GPT-5.5 intelligence at half the price.
• Provider-side efficiency shows up in list prices at their discretion. Architecture-side efficiency shows up in your COGS immediately.
• The expensive part of an agent product is the repeated region of a single turn, not the single model call you benchmarked.
• Model the effect on your tiers and margins before you reprice, because customer expectations reset faster than your cost base does.

What did OpenAI actually change?

Three layers, not one. The model itself is trained to do more work per token. The inference stack was reworked: load balancing, speculative decoding, prompt caching, and GPU kernels, with the reported result being roughly 20% lower end-to-end serving cost from kernel work and more than 15% better token-generation efficiency from the improved draft model. The third layer is the agentic harness, a Rust orchestration layer that caps tool output at 10,000 tokens by default and keeps model-visible history append-only so prompt-cache prefixes stay intact.

That third layer is the part worth copying. The first two are OpenAI's cost problem. The third is yours.

Why do cheaper tokens not automatically raise your margin?

Because a provider's efficiency gain is an input to their pricing decision, not a rebate to your invoice. You benefit only when it is passed through as a lower price per million tokens, on the specific tier and modality you actually use, and only in proportion to how much of your bill sits in that line item.

Say, illustratively, that inference is 55% of your COGS and half of that spend sits on a tier that gets a 20% price cut. Your COGS falls about 5.5%. On a 60% gross margin that is a couple of points, not a re-rating. Meanwhile, if your agent loop grew from 8 model calls to 30 over the same two quarters, you gave the entire gain back and then some.

Where does the cost actually live in an agent product?

Inside the loop. One user turn can contain many model calls and tool calls, and anything sitting in that repeated region gets paid for repeatedly: the system prompt, the tool definitions, the conversation history, the last tool result. This is the number most founders never measure, because their spreadsheet models one request and their product ships thirty.

Two cheap architectural fixes follow directly from the OpenAI write-up:

1. 1Keep prefixes byte-stable. Append new messages and tool results at the end. Never insert into earlier context, never reorder tool definitions, never bake runtime settings into tool schemas. Every prefix mutation is a cache miss you pay full price for.
2. 2Cap what tools can return. A tool that dumps 40,000 tokens of log output into context does not just cost you once. It costs you on every subsequent iteration of that turn.

How should this change your pricing, not just your infrastructure?

Carefully, and in that order. Falling token prices are a trap for anyone whose plan pricing is implicitly indexed to them. Two failure modes are common: the seat-based product that quietly funds unlimited agent usage on its highest-margin plan, and the usage-based product that passes cost cuts through so fast it never builds a margin buffer for the next capability jump.

A better sequence: measure cost per successful task, not cost per token. Segment it by tier and by customer cohort. Then decide how much of a provider price cut you keep as margin, how much you spend on capability (longer loops, better models, more retries), and how much you pass to customers as a competitive move. Those are three different decisions and they should be made explicitly.

One more thing worth noticing

The post describes GPT-5.6 Sol doing much of its own optimization work: rewriting production kernels, running hundreds of draft-model experiments, tuning serving configurations. If that pattern holds, provider cost curves start improving on a compute schedule rather than an engineering-headcount schedule. Plan your pricing on the assumption that per-token costs keep falling and that your competitors' costs fall too. Durable margin comes from what you build around the model, not from the model's price tag.

Takeaway: treat every provider price cut as a decision to allocate, not a windfall to celebrate. You can model the tier-by-tier margin effect in Calcaas before you touch your pricing page.