vLLM vs TensorRT-LLM: The 12% Cost Gap and When It Is Worth Paying For
On the same H100, TensorRT-LLM serves a million output tokens for about $0.66 against vLLM's $0.75, and whether that 12% is worth having depends almost entirely on how often you change models.
Key takeaways
- At 50 concurrent requests on a single H100 SXM5 serving Llama 3.3 70B at FP8, TensorRT-LLM sustained 2,100 tokens/sec against vLLM's 1,850, a 13% edge that widens to 16% at 100 concurrent requests.
- At $5.01/hr on-demand, that works out to roughly $0.66 per million output tokens for TensorRT-LLM against $0.75 for vLLM.
- TensorRT-LLM took about 28 minutes to compile a Llama 3.3 70B FP8 engine. vLLM went from cold container to first served request in about 62 seconds.
- TTFT p95 at 100 concurrent requests was 1,280ms for TensorRT-LLM against 1,450ms for vLLM, a 170ms gap that shrinks to single-digit milliseconds at low concurrency.
- TensorRT-LLM runs on NVIDIA only. vLLM supports NVIDIA, AMD ROCm, Intel, Google TPU, AWS Neuron, and ARM.
How large is the throughput gap?
Both engines were run on the same single H100 SXM5 80GB bare-metal instance serving meta-llama/Llama-3.3-70B-Instruct at FP8, with 512 average input tokens and 256 average output tokens across 200 prompts.
| Concurrency | vLLM | TensorRT-LLM | Advantage | | --- | --- | --- | --- | | 1 req | 120 tok/s | 130 tok/s | +8% | | 10 req | 650 tok/s | 710 tok/s | +9% | | 50 req | 1,850 tok/s | 2,100 tok/s | +13% | | 100 req | 2,400 tok/s | 2,780 tok/s | +16% |
The compiled kernel graph does its best work exactly where GPU scheduling gets hardest, at high batch occupancy. At low concurrency the gap is small enough that it should not drive your decision.
What is the cost per million tokens?
Cost per token is throughput divided into the GPU hourly rate. At $5.01/hr for an H100 SXM5 on-demand, using the 50-concurrent numbers:
| Engine | Throughput | Cost per 1M output tokens | | --- | --- | --- | | vLLM | 1,850 tok/s | $0.75 | | TensorRT-LLM | 2,100 tok/s | $0.66 |
So roughly 12% cheaper output tokens on identical hardware, widening as concurrency climbs toward 100 requests.
When does that 12% actually pay for itself?
Here is the calculation almost nobody runs, and it is the whole decision.
The compile is a fixed cost per model version, not per request. Compiling a Llama 3.3 70B FP8 engine took about 28 minutes on a single H100. At $5.01/hr that is roughly $2.34 of GPU time, which is trivial. The real cost is the pipeline you now have to build and maintain around it, and the operational constraints it imposes.
So amortise it. Take your monthly output-token volume, multiply by the $0.09 per million saving, and compare that against the engineering cost of maintaining a quantize-then-compile pipeline plus the friction it adds to every deploy.
The friction is where it bites. Blue-green deploys, scale-to-zero autoscaling, and frequent model swaps all get harder when every new model version needs a 28-minute build before it can take traffic. A team running one stable model for months absorbs that cost once and forgets about it. A team shipping model updates weekly pays it every week, in calendar time and in deploy risk, to save nine cents per million tokens.
The honest framing: your model-change frequency is the deciding variable, not your throughput requirement. Most engine comparisons never mention it.
What about latency and hardware lock-in?
On latency, TensorRT-LLM's p95 TTFT at 100 concurrent requests was 1,280ms against vLLM's 1,450ms. That 170ms gap is noticeable in an interactive product under load. At a single request the difference was 38ms against 45ms, which is not.
On hardware, the asymmetry is structural. TensorRT-LLM supports NVIDIA GPUs only, on Linux x86_64 or aarch64, because it is built as a compiler around NVIDIA's own toolchain. vLLM supports NVIDIA CUDA, AMD ROCm, Intel GPUs, Google TPU, AWS Neuron, and ARM, and has been a PyTorch Foundation project since May 2025.
If every GPU you will ever run is NVIDIA, that is not a differentiator. If there is any chance you diversify hardware later, for cost or for capacity reasons, one option keeps that door open and the other closes it. Worth noting that TensorRT-LLM v1.0 and later ship a PyTorch backend, now the default, that loads Hugging Face weights directly and skips the compiled-engine step, trading some peak throughput for a vLLM-like cold start.
The one line to keep: the engine gap is 12%, the compile tax is fixed per model version, and your release cadence decides which one dominates.
You can put both cost-per-token figures into your own volume model at calcaas.com.
Frequently asked questions
Which engine is cheaper per million tokens, vLLM or TensorRT-LLM?
At sustained high concurrency, TensorRT-LLM. On a single H100 SXM5 at $5.01/hr serving Llama 3.3 70B at FP8, TensorRT-LLM's 2,100 tokens/sec works out to roughly $0.66 per million output tokens against vLLM's 1,850 tokens/sec at $0.75, about 12% cheaper.
How much faster is TensorRT-LLM than vLLM?
Between 8% and 16% depending on load. On the same H100 benchmark it led at every concurrency level tested: 8% faster at a single request, 13% at 50 concurrent requests, and 16% at 100. The gap widens as batch occupancy increases.
How long does TensorRT-LLM take to compile an engine?
About 28 minutes for a Llama 3.3 70B FP8 engine on a single H100, a one-time cost per model version that is reused on every restart, with a saved engine reloading in roughly 90 seconds. vLLM has no compile step and serves its first request about 62 seconds from a cold container.
Does TensorRT-LLM run on AMD GPUs or TPUs?
No. TensorRT-LLM runs on NVIDIA GPUs only, on Linux x86_64 or aarch64. vLLM supports NVIDIA CUDA plus AMD ROCm, Intel GPUs, Google TPU, AWS Neuron, and ARM, which matters if you want to avoid being locked to one vendor's hardware roadmap. Place the JSON-LD above inside a <script type="application/ld+json"> tag in the page head.
More from the blog
LLM EconomicsAI Agent Hosting Cost: Price Per Completed Task, Not Per GPU-Hour
On the same GPU at the same hourly rate, a ten-step agent loop costs roughly 14x a single chat call per unit of finished work, and most of that gap is context you re-send rather than work you do.
LLM EconomicsWhy a GPU Cloud Rate Card Is a Ceiling, Not a Price
CoreWeave's published H100 rate is $6.16 per GPU-hour, but the best committed discount it advertises lands in the same band as a no-commitment marketplace rate, which means a multi-year contract buys capacity certainty rather than savings.
LLM EconomicsH100 vs A100 Cost: Why a Faster GPU Can Still Raise Your Bill
An H100 SXM5 costs about 2.75x more per hour than an A100 80GB SXM4, so it only saves money when its throughput advantage clears that same ratio.
Pricing math, in your inbox.
One short note a week on AI pricing, token economics, and margin. No spam, unsubscribe anytime.