Inference Engineering Is a Gross Margin Lever, Not an Infra Detail

Key takeaways

• Inference optimizations reported in the Latent Space masterclass with Baseten produce gains of 20%, 100%, and in some cases 200% on the same hardware.
• In one GLM-5.2 experiment, quantizing more of the model preserved benchmark quality while increasing throughput by roughly 20%, because errors introduced in different layers partly cancelled out.
• Baseten raised a $13B Series F, which is the market pricing inference as a category rather than a feature.
• If you buy tokens from an API, these gains show up in your provider's margin, not yours, unless they cut the price.
• The founder decision is not 'which model', it is 'which side of the margin curve do I want to sit on'.

What is inference engineering, in plain terms?

Three years ago it barely existed as a job title. Today it answers one question: how do you turn weights from training into a product that is fast, reliable and affordable at scale?

In the Latent Space episode, Baseten's Philip Kiely and Ali Taha walk through the levers: cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV cache movement, model parallelism and GPU kernels. If you are not serving models yourself, none of that is your problem. But all of it is in your bill.

Why does a 20% throughput gain matter to a founder?

Because throughput and cost are the same number viewed from different ends.

A GPU costs you a fixed amount per hour whether it produces 100 tokens per second or 200. Double the tokens per second and you halve the cost per token. That is the whole trick, and it is why the numbers in this episode are worth reading as financial figures rather than engineering figures.

The quantization result is the one to sit with. Quantizing more of GLM-5.2 raised throughput by about 20% without losing benchmark quality, because errors introduced in one layer partially cancelled errors in another. Roughly a fifth off the serving cost for the same output quality.

Here is the part the episode does not say out loud, and it is the part that decides your P&L: that 20% belongs to whoever owns the GPU.

If you rent tokens from an API, your provider banks the gain. You see it only if competition forces them to cut list prices. If you own the serving stack, it flows straight into gross margin. Same engineering result, opposite balance sheets.

What actually drives cost on a long request?

Ask what happens when a 200,000-token request enters an inference system and the cost structure becomes obvious.

• Prefill processes the whole prompt. It is compute-heavy and scales with context length.
• Decode generates tokens one at a time. It is memory-bandwidth-heavy and scales with output length.

These two have such different profiles that serious stacks now run them on different GPUs, which is what 'disaggregated prefill and decode' means. Cache-aware routing then sends a follow-up turn to the machine that already holds the relevant KV cache, so the system skips recomputing context it has already seen.

The practical implication for pricing: long context is not linearly more expensive, it is differently expensive. A product where users paste large documents has a cost curve dominated by prefill. A product that generates long outputs is dominated by decode. Those two products should not have the same pricing model, and most of the time they do.

So should you self-host?

Usually not on day one. Here is the honest trade:

| | Buy tokens from an API | Serve it yourself | |---|---|---| | Cost shape | Variable, per token | Fixed, per GPU hour | | Who captures throughput gains | Provider | You | | Engineering cost | Near zero | An inference engineer, at least | | Risk at low volume | None | Idle GPUs | | Risk at high volume | Margin compression | Capacity planning |

The crossover is about volume predictability, not volume alone. Spiky traffic on owned hardware means you pay for the peak and use the average. Steady traffic concentrated on one or two models is where owning the stack starts paying.

A useful illustrative check: say your token spend is $8,000 a month across a handful of models. One inference engineer costs several times that. Self-hosting is not a saving at that scale, it is a hobby. Say it is $180,000 a month on one model with flat weekday traffic: now a 20% serving gain is worth $36,000 a month and the maths inverts.

One line to take away: inference engineering is where AI gross margin is actually made, so decide deliberately whether you are buying that margin or building it.

If you want to see where your own crossover point sits, you can model token volumes, provider rates and per-tier margins side by side in Calcaas before you commit to either path.