GPU Cost Per Million Tokens in 2026: What Self-Hosting an LLM Really Costs

Key takeaways

• Spheron's 2026 benchmarks (vLLM, batch 256): Llama 4 Scout 17B at ~$0.16-0.19 per 1M tokens on a single A100/H100; Llama 3.3 70B at ~$2.30 on 8x H100; DeepSeek V3 at ~$3.58.
• One formula does all the work: CPM = (cluster $/hr) / (tokens per second x 3600 / 1,000,000).
• FP8 quantization roughly halves cost per token on H100-class hardware, with reported quality loss under 2% on standard benchmarks.
• Batch size is the hidden multiplier: batch 1 vs batch 256 on the same cluster is a ~100x cost-per-token difference.
• These numbers assume near-saturated hardware; at typical startup volumes, API pricing usually wins below roughly 50-100M tokens per month.

How do you calculate cost per million tokens?

Divide what the hardware costs per hour by how many millions of tokens it produces per hour:

CPM = (cluster $/hr) / (tokenspersec x 3600 / 1,000,000)

Worked example from the benchmark: a single H100 SXM5 at $2.90/hr running Llama 4 Scout 17B at 4,200 tokens/sec produces about 15.1M tokens per hour, so the cost lands near $0.19 per million tokens. The same model on a cheaper A100 at $1.64/hr and 2,800 tok/s comes out around $0.16. Hourly rate alone told you nothing: the A100 is "slower" and still cheaper per token here.

What do the 2026 benchmarks actually show?

Selected results (vLLM, continuous batching, batch 256, FP16 unless noted, April 2026 on-demand pricing):

| Model | Config | Cost per 1M tokens | | --- | --- | --- | | Llama 4 Scout 17B (MoE) | 1x A100 80G | ~$0.16 | | Qwen 3 30B-A3B (MoE) | 1x H100 | ~$0.21 | | Gemma 4 31B | 1x H100 | ~$0.26 | | Mistral Small 3.1 24B | 2x H100 | ~$0.47 | | Llama 3.3 70B | 8x H100 | ~$2.30 (FP8: ~$1.15) | | Llama 3.3 70B | 8x B200 (spot) | ~$0.88 | | DeepSeek V3 671B (MoE) | 8x H100 | ~$3.58 |

Two patterns stand out. Small mixture-of-experts models are extraordinarily cheap to serve: active parameters, not total parameters, drive throughput. And very large models pay twice: more GPUs in the numerator and fewer tokens per second in the denominator.

Why is quantization the cheapest upgrade available?

Moving Llama 3.3 70B from FP16 to FP8 on the same 8x H100 cluster roughly doubles throughput, which halves the cost from ~$2.30 to ~$1.15 per million tokens, with reported quality loss under 1-2% on standard benchmarks. INT4 pushes it to ~$0.77 with a larger, task-dependent quality tradeoff. If you serve open models and have not tested FP8, you are likely paying about double for no measured benefit.

Why does batch size matter more than GPU choice?

At batch size 1, the benchmark's 8x H100 cluster works out to roughly $258 per million tokens. At batch 256 with continuous batching, $2.30. Same hardware, ~100x difference, because a GPU serving one request at a time spends most of its life waiting on memory instead of generating.

The uncomfortable middle: many production teams run at effective batch sizes of 8-32 due to under-configured serving, which leaves them 10-30x above the cost they think they benchmarked.

When does self-hosting actually beat API pricing?

Here is the observation the benchmark tables skip: every figure above assumes a queue of traffic keeping the hardware busy. A startup serving scattered daytime traffic might average 20% utilization, which quietly multiplies its real cost per token by 5x. The sticker number is a best case, not a budget.

As a rule of thumb from the source's companion analysis: below roughly 20M tokens per month, managed APIs win once you count ops; above roughly 100M with steady traffic, self-hosting usually wins. In between, it depends on traffic shape, latency needs, and your team.

The one-line takeaway: cost per million tokens is a function of utilization first and hardware second. To compare self-hosted cost against API tier pricing for your actual traffic, you can run both scenarios side by side in Calcaas.