Distillation Just Got 15x Cheaper. Here Is What That Does to Your AI Cost Model

Key takeaways

• In an isolated benchmark from Multiverse Computing, peak memory at 32K tokens fell from 85.2 GiB with a dense KL loss to 5.45 GiB with a fused chunked loss, a 15.6x reduction.
• A GPT-OSS 20B distillation run at 32,768-token context shrank from four GPU nodes to one. Step time dropped from 57.0 seconds to 12.23 seconds, roughly 5x faster.
• The dense baseline fails outright from 64K tokens onward, so long-context distillation was previously gated on cluster access rather than on ideas.
• Distillation is a fixed cost. Per-token API pricing is a variable cost. Comparing the two requires a volume assumption, and most teams never write that assumption down.
• The cost that actually moved here is the cost of trying. Cheap experiments change which teams can afford to find out whether a small model works for them.

What actually changed?

Knowledge distillation trains a smaller student model to imitate a larger teacher. The expensive part has never been the idea, it has been the memory. The standard setup keeps both models loaded and builds a probability distribution over the entire vocabulary for every token position. For a model with a 201,088-token vocabulary at 32K sequence length and batch size 4, that single tensor is about 50GB in bfloat16, and a full training iteration can peak near 250GB.

The research from Multiverse Computing makes two systems changes. First, the teacher's top-100 logits are computed once and cached, so the teacher never has to sit in memory beside the student. Second, a fused chunked KL loss folds the model's output projection into the loss and processes one slice of the sequence at a time, so the full vocabulary-by-sequence grid is never built.

Why does a memory number matter for pricing?

Because memory is the gate on cluster size, and cluster size is the bill.

On a single H200 at 8K context, the measured difference between online distillation and the fused chunked offline version is 102.8 GB peak memory versus 58.3 GB. That is real but unremarkable. The number that matters commercially shows up at long context: at 32K tokens the isolated benchmark drops from 85.2 GiB to 5.45 GiB, and the practical GPT-OSS 20B run went from four nodes to one.

A four-to-one node reduction is not a 4x saving on one training run. It is a change in who can run the experiment at all.

How do you decide between distilling and calling an API?

Write both sides as a total monthly cost and find where the lines cross.

The API side is simple: requests per month, multiplied by tokens per request, multiplied by the provider's input and output rates. It is linear from zero, with no upfront commitment.

The self-hosted side has two parts most cost models omit. There is a one-time cost to produce the model, distillation compute plus the engineering time to run it, and a recurring cost to serve it, GPU hours whether or not traffic arrives. That second part is why self-hosting is expensive at low volume: you pay for idle capacity that an API charges you nothing for.

The crossover is the whole decision. Below it, an API is cheaper and you should not be reading distillation papers. Above it, the fixed cost amortises and every additional request widens the gap.

What number moves when training gets cheaper?

The fixed cost, which lowers the crossover point. Cheaper distillation does not make self-hosting cheaper per request, it makes the entry ticket smaller. A team that needed a four-node budget to even attempt a compressed model can now attempt it on one node and iterate.

Does the smaller model still work?

In this case, mostly. The resulting student, distilled from Llama 3.1 8B Instruct down to about 3.2B parameters, retains most of the teacher's accuracy on BoolQ and HellaSwag and stays within about nine points on MMLU, at less than half the parameter count. Nine points on MMLU is not nothing. Whether it is acceptable is a product question, not a cost question, and it should be answered before any spreadsheet is opened.

The observation most cost models miss

Almost every AI cost model treats model choice as a fixed input and token volume as the variable. That is backwards for anyone considering compression. The real variable is how many attempts you can afford before you find a student model that holds quality. At four nodes per attempt, most teams get one shot and conclude it does not work. At one node per attempt, they get several, and the conclusion changes.

Cheap experimentation is a cost lever that never appears on a per-token price list.

One-line takeaway: distillation research just lowered the fixed cost of owning a model, which means the volume at which owning beats renting is lower than your last spreadsheet said it was.

If you want to see where that crossover sits for your own traffic, model your per-request and per-user token cost in the Calcaas LLM cost calculator.