AI Agent Hosting Cost: Price Per Completed Task, Not Per GPU-Hour
On the same GPU at the same hourly rate, a ten-step agent loop costs roughly 14x a single chat call per unit of finished work, and most of that gap is context you re-send rather than work you do.
Key takeaways
- Agentic workflows typically chain 10 to 50 LLM calls to finish one task, which researchers estimate multiplies token consumption 5 to 30x over a standard chatbot exchange.
- Re-sent context, meaning the repeated system prompt, tool definitions, and conversation history, accounts for 62% of total agent inference spend according to research cited by Cockroach Labs drawing on Stanford's Digital Economy Lab.
- In a benchmark on a single H100 SXM5 at $3.92/hr, a single chat call costs about $0.027 per 1,000 calls, while a ten-step agent loop costs about $0.38 per 1,000 completed tasks.
- Neocloud on-demand pricing runs 2 to 5x below AWS and Azure for the same H100, and that gap compounds across every step of an agent loop.
- The founder trap is shipping an agent feature inside a per-seat price that was costed for a chat-shaped workload.
Why does an agent task cost so much more than a chat call?
Because an agent loop calls the model again after every tool invocation. One conversation might hit the GPU five, ten, or fifty times before the user sees a final answer, and every one of those calls carries the system prompt, the full tool schema list, and the accumulated results of every prior step.
That re-sent baggage is the single largest line item. The model is not generating it, it is re-reading it on every step, and it accounts for 62% of total agent inference spend.
Sit with that number for a second. The majority of your agent bill is redundancy, not work. Which means the highest-leverage cost lever is prefix caching and context discipline, not GPU selection.
What does the arithmetic look like on real hardware?
A benchmark on a single H100 SXM5 running Llama 3.1 8B Instruct at FP8 with vLLM continuous batching sustained 22,400 output tokens per second across 100 concurrent sessions. At the live on-demand rate of $3.92/hr, that works out to roughly $0.0000000486 per token.
From there:
- Single chat call, roughly 300 input plus 250 output tokens: about $0.000027 per call, or $0.027 per 1,000 calls.
- Ten-step agent loop for one completed task, roughly 7,800 tokens once you account for re-sent context: about $0.00038 per task, or $0.38 per 1,000 completed tasks.
Same GPU. Same hourly rate. Same cost per token. About 14x more per unit of finished work.
This is why picking a provider on dollars per hour misses the number that actually determines your bill. Two teams on identical hardware at identical rates can see wildly different costs per completed task, purely from how aggressively their loop re-sends context.
Which GPU tier should each agent run on?
Tiering compute to the task is the highest-leverage lever available before you touch the loop itself:
| Tier | Workload | On-demand rate | | --- | --- | --- | | L40S 48GB | Routing, intent classification, lightweight 7-8B models | ~$0.96/hr | | H100 PCIe 80GB | Production user-facing agent APIs, 13B-class models | ~$2.65/hr | | H200 SXM5 141GB | Reasoning-heavy agents, 32K+ context, long KV caches | ~$4.22/hr |
Do not run a routing model that handles 95% of your requests on H100 pricing. And note the provider gap: the same H100 runs roughly $6.88/hr on AWS and roughly $12.29/hr on Azure, so a 2 to 5x structural premium multiplies across every step of every loop.
What breaks first under agent load?
VRAM, not compute. KV cache exhaustion is usually the first failure mode, because each tool call pauses the model mid-reasoning while the tool executes and the session's state sits in memory for the full pause. A multi-turn agent conversation accumulates KV cache with every turn, so sizing for parameter count alone under-provisions the moment a real conversation runs past two or three turns.
On latency, a reasonable production target for synchronous agents is TTFT P95 under 500ms and P99 under 800ms, with voice agents needing the LLM component under 200ms. The 100-concurrent-session benchmark above held TTFT P50 at 78ms and P99 at 342ms at 84% average GPU utilization, comfortably inside those budgets.
One operational rule that is not negotiable: if a real user is waiting on the response, do not put it on spot. Spot belongs on background research pipelines, async enrichment, and scheduled summarization that can checkpoint and resume.
What should a founder actually do about it?
The risk that catches teams is not GPU choice, it is pricing. If you shipped an agent feature into a per-seat tier that was costed against a chat-shaped workload, your COGS per active user may have moved by an order of magnitude while your price did not.
Gartner predicts 40% of enterprise applications will feature task-specific AI agents by the end of 2026, up from under 5% in 2025, and separately that over 40% of agentic AI projects will be cancelled by the end of 2027, citing escalating costs among the top reasons. Those two forecasts are the same story told twice.
So: instrument cost per completed task before you launch, re-price the tier that contains the agent feature, and treat prefix caching as a revenue project rather than an optimization.
The short version: GPU-hours are what you buy, completed tasks are what you sell, and only one of those belongs in your pricing page.
You can compare GPU tiers at your own concurrency and token profile at calcaas.com.
Frequently asked questions
Why does AI agent hosting cost more than chat hosting?
Because an agent loop calls the model repeatedly to complete one task, and each call re-sends the system prompt, tool schemas, and every prior tool result. Agentic workflows typically chain 10 to 50 calls per completed task, which researchers estimate consumes 5 to 30x more tokens than a single chatbot exchange.
How much of an agent's cost is wasted on repeated context?
About 62%. Research cited by Cockroach Labs, drawing on Stanford's Digital Economy Lab, found that re-sent context, meaning the repeated system prompt, tool definitions, and conversation history, accounts for 62% of total agent inference spend.
What is the real cost difference between a chat call and an agent task?
On a single H100 SXM5 at $3.92/hr, a chat call of roughly 550 tokens costs about $0.027 per 1,000 calls, while a ten-step agent loop of roughly 7,800 tokens costs about $0.38 per 1,000 completed tasks. That is about 14x more per unit of finished work on identical hardware.
Which GPU should I use for AI agents?
Tier it to the agent. Routing and classification agents run comfortably on an L40S at roughly $0.96/hr, production user-facing agent APIs suit an H100 PCIe at roughly $2.65/hr, and reasoning-heavy agents with 32K or more context need an H200's 141GB at roughly $4.22/hr to keep the KV cache resident. Place the JSON-LD above inside a <script type="application/ld+json"> tag in the page head.
More from the blog
LLM EconomicsvLLM vs TensorRT-LLM: The 12% Cost Gap and When It Is Worth Paying For
On the same H100, TensorRT-LLM serves a million output tokens for about $0.66 against vLLM's $0.75, and whether that 12% is worth having depends almost entirely on how often you change models.
LLM EconomicsWhy a GPU Cloud Rate Card Is a Ceiling, Not a Price
CoreWeave's published H100 rate is $6.16 per GPU-hour, but the best committed discount it advertises lands in the same band as a no-commitment marketplace rate, which means a multi-year contract buys capacity certainty rather than savings.
LLM EconomicsH100 vs A100 Cost: Why a Faster GPU Can Still Raise Your Bill
An H100 SXM5 costs about 2.75x more per hour than an A100 80GB SXM4, so it only saves money when its throughput advantage clears that same ratio.
Pricing math, in your inbox.
One short note a week on AI pricing, token economics, and margin. No spam, unsubscribe anytime.