Why the GPU-Hour Sticker Price Isn't What You'll Pay for AI Inference at Scale

Key takeaways

• A recent snapshot of GPU rental listings on Vast.ai found that headline per-GPU-hour prices barely move as cluster size increases, even though the supply of machines that can actually fill the request shrinks fast.
• Needing four co-located H200s cut eligible supply in half while the price measure rose only 4% (from $3.93 to $4.08 per GPU-hour); needing eight left zero qualifying supply in the sample.
• A100 GPUs showed the opposite pattern: the price measure rose 114%, from $0.60 to $1.29 per GPU-hour, as the requested cluster grew from one GPU to eight.
• The same gap between sticker price and the price you can actually get shows up in LLM API pricing, where an advertised dollar-per-million-token rate assumes you can obtain your needed throughput at that rate.
• Modeling margins off headline per-token rates instead of throughput-adjusted cost is a common way founders overestimate their true unit economics.

What did the GPU rental data actually show?

A survey of on-demand offers on Vast.ai, the GPU rental marketplace, filtered down to 37 physical machines and 93 listed GPUs across five accelerator models (A100 SXM4, H100 SXM, H200, B200 and L40S), keeping only listings that were verified, currently rentable, available for at least seven days, and hosted at 99%+ reliability.

For H200, going from a one-GPU to a two-GPU request left 92% of eligible inventory intact at essentially the same price. Going to four GPUs cut eligible inventory to half, with the price measure (the median of the three cheapest eligible machines) rising just 4%. At eight GPUs, nothing in the sample qualified, at any price.

H100 and B200 supply was thinner still: H100 supported a two-GPU request but not a four-GPU one, and the B200 sample (two physical machines) topped out at two GPUs each, so four- and eight-GPU availability was zero. A100 behaved the way a textbook would predict: price rose as supply thinned. L40S cut both ways, with only 35% of listed inventory sitting in machines big enough for an eight-GPU request, though the one qualifying machine happened to be cheap.

Why does compute ration through configuration instead of price?

In a normal commodity market, scarcity shows up as a higher price. Compute mostly doesn't work that way. A marketplace can show plenty of aggregate GPU-hours while a buyer still can't assemble them into one working cluster, because the binding constraints (hardware generation, co-location, interconnect, reliability, reservation length, geography) ration supply before price has a chance to move. When no configuration exists that meets a buyer's requirement, the trade simply doesn't happen, so the scarcity never appears in the index at all. That's why an index built from one- and two-GPU prices can make compute look abundant and stable, even while a team trying to assemble a large cluster can't find one at any price.

How does this apply to LLM API pricing, not just GPU rental?

The same pattern hides inside inference API pricing pages. A provider's advertised rate, say a certain number of dollars per million output tokens, is a headline number for a single request under normal conditions. It doesn't say what happens once you need that rate sustained at your real concurrency: past a certain volume you may hit rate limits, get pushed toward batch processing with longer latency, or need a reserved-throughput or dedicated-capacity agreement priced differently from the public rate card. Just like the GPU cluster buyer, you can be quoted a clean per-unit price and still not be able to buy the quantity you actually need at that price.

What should this change about how you model AI margins?

If you're pricing a product around a provider's headline dollar-per-million-token rate, you're implicitly assuming your real-world throughput costs the same as a single light-traffic request. That assumption tends to break exactly when a product scales, which is also exactly when margin mistakes get expensive. A more durable model separates the sticker rate from the effective rate you'd actually pay at your projected concurrency and volume, the same way a GPU buyer should track how many machines still qualify at the cluster size they need, not just the quoted hourly rate.

Cluster size, and its LLM equivalent, sustained throughput, belongs in the cost basis, not just the benchmark. If you want to see how sensitive your margins are to that gap, you can model both the headline and throughput-adjusted cost scenarios side by side in Calcaas.