H100 vs A100 Cost: Why a Faster GPU Can Still Raise Your Bill
An H100 SXM5 costs about 2.75x more per hour than an A100 80GB SXM4, so it only saves money when its throughput advantage clears that same ratio.
Key takeaways
- Live rates put H100 SXM5 at roughly $5.01/hr on-demand and A100 80GB SXM4 at roughly $1.82/hr, a 2.75x gap. Spot narrows it to about $2.91 against $1.15.
- On a fixed 10 billion token fine-tune, an 8x H100 cluster running BF16 finishes in half the time of 8x A100 and still costs about 30% more: roughly $2,930 against $2,246.
- Turn on FP8 through the Transformer Engine and the same H100 cluster drops to roughly $1,796, which is about 20% cheaper than A100 and 3.4x faster.
- For inference, NVIDIA's GPT-J 6B benchmark at batch 64 shows H100 FP8 at 10,907 tokens/sec against A100 FP16's 3,679, a 3.0x gap that just clears the 2.75x price premium.
- The decision rule is one line: compare the throughput ratio to the price ratio, and never look at either in isolation.
What is the actual price gap?
Rates as of 12 August 2026 on Spheron's marketplace:
| GPU | On-demand $/hr | Spot $/hr | | --- | --- | --- | | H100 SXM5 | ~$5.01 | ~$2.91 | | H100 PCIe | ~$2.65 | ~$2.20 | | H100 NVL | ~$2.75 | ~$2.60 | | A100 80GB SXM4 | ~$1.82 | ~$1.15 | | A100 80GB PCIe | ~$1.48 | ~$1.19 |
Notice how much the form factor moves the comparison. H100 PCIe against A100 SXM4 is under a 1.5x premium instead of 2.75x, which lowers the throughput bar the H100 has to clear. If your workload does not need NVLink bandwidth, comparing SXM5 to A100 is comparing the wrong two products.
Why is FP8 the whole argument?
Three architectural differences drive nearly all of the performance gap: memory bandwidth (3.35 TB/s HBM3 against 2.0 TB/s HBM2e), NVLink bandwidth (900 GB/s against 600 GB/s), and the Transformer Engine's native FP8 support, which A100 lacks entirely.
That last one is the swing factor. BF16 dense tensor throughput is 989.4 TFLOPS on H100 against 312 on A100 on paper, but if your stack never enables FP8, you are paying for Hopper silicon you are not using.
Here is what that looks like as money, on a fixed 10 billion token fine-tune:
| Cluster | Precision | Tok/s | Wall time | Cluster $/hr | Total cost | | --- | --- | --- | --- | --- | --- | | 8x A100 SXM4 | BF16 | 18,000 | ~154 hrs | $14.56 | ~$2,246 | | 8x H100 SXM5 | BF16 | 38,000 | ~73 hrs | $40.08 | ~$2,930 | | 8x H100 SXM5 | FP8 | 62,000 | ~45 hrs | $40.08 | ~$1,796 |
The middle row is the trap, and it is worth naming clearly. That configuration is genuinely twice as fast and 30% more expensive. It looks like a performance upgrade on every dashboard your team watches, and it is a cost regression on the only dashboard finance watches.
Which GPU is cheaper per token for inference?
Closer than most people assume. On NVIDIA's TensorRT-LLM benchmark for GPT-J 6B at batch size 64, H100 FP8 delivers 10,907 tokens/sec against A100 FP16's 3,679, a 3.0x gap. Run that through live pricing and cost per million tokens lands at roughly $0.13 for H100 and $0.14 for A100.
That is a rounding error, and it only goes H100's way because the 3.0x throughput edge barely clears the 2.75x price gap. Drop the batch size, or run A100 in BF16 without quantization, and the winner flips.
There is a second finding worth attention. SemiAnalysis's InferenceX measurements put H100 serving GPT-OSS-120B at around $0.13 per million tokens at 117 tokens/sec per user, rising to $0.44 per million at 216 tokens/sec per user. Your cost per token more than triples because you promised users a faster experience.
That reframes something founders usually treat as a product decision: your latency SLO is a pricing input. If you commit to a high per-user token rate in your product, you have committed to a materially higher COGS, and that belongs in the model before it goes on the marketing page.
When is A100 still the right call?
- 7B to 13B LoRA or QLoRA fine-tunes, where jobs finish in hours and hourly rate outweighs peak throughput.
- BF16-only pipelines where FP8 is not wired up, which removes H100's structural advantage.
- Legacy CUDA stacks without Hopper-tuned kernels.
- Spot-tolerant batch workloads, where A100 spot around $1.15/hr produces the lowest cost per token of any option here.
H100 wins when FP8 is genuinely exercised, when you are pretraining at 70B and above, when you serve at consistently high concurrency, or when wall-clock time itself carries a cost.
The one line to remember: a hardware upgrade is a cost win only when the throughput ratio exceeds the price ratio, and a latency promise is a cost commitment.
You can run the same ratio test on your own workload at calcaas.com.
Frequently asked questions
Is the H100 cheaper per token than the A100?
Only marginally, and only in the right configuration. On NVIDIA's GPT-J 6B benchmark at batch 64, H100 FP8 reaches 10,907 tokens/sec against A100 FP16's 3,679, which works out to roughly $0.13 per million tokens against $0.14. At lower batch sizes or without FP8, A100 wins.
How much more does an H100 cost per hour than an A100?
About 2.75x. Live rates put H100 SXM5 at roughly $5.01/hr on-demand against A100 80GB SXM4 at roughly $1.82/hr. Spot narrows it slightly, around $2.91 against $1.15, and H100 PCIe at roughly $2.65/hr shrinks the premium to under 1.5x.
Why can an H100 cluster cost more to finish the same training job?
Because a 2.1x throughput gain does not cover a 2.75x hourly premium. On a fixed 10 billion token fine-tune, 8x H100 in BF16 finishes in about half the time of 8x A100 but costs roughly $2,930 against $2,246. Enabling FP8 raises throughput to 3.4x and drops the cost to about $1,796.
Does my latency target affect cost per token?
Significantly. SemiAnalysis's InferenceX measurements put H100 serving GPT-OSS-120B at around $0.13 per million tokens at 117 tokens/sec per user, rising to $0.44 per million at 216 tokens/sec per user. A faster per-user experience is a direct increase in cost of goods sold. Place the JSON-LD above inside a <script type="application/ld+json"> tag in the page head.
More from the blog
LLM EconomicsvLLM vs TensorRT-LLM: The 12% Cost Gap and When It Is Worth Paying For
On the same H100, TensorRT-LLM serves a million output tokens for about $0.66 against vLLM's $0.75, and whether that 12% is worth having depends almost entirely on how often you change models.
LLM EconomicsAI Agent Hosting Cost: Price Per Completed Task, Not Per GPU-Hour
On the same GPU at the same hourly rate, a ten-step agent loop costs roughly 14x a single chat call per unit of finished work, and most of that gap is context you re-send rather than work you do.
LLM EconomicsWhy a GPU Cloud Rate Card Is a Ceiling, Not a Price
CoreWeave's published H100 rate is $6.16 per GPU-hour, but the best committed discount it advertises lands in the same band as a no-commitment marketplace rate, which means a multi-year contract buys capacity certainty rather than savings.
Pricing math, in your inbox.
One short note a week on AI pricing, token economics, and margin. No spam, unsubscribe anytime.