Idle GPUs Are the Most Expensive Line in Your AI Budget

Key takeaways

• A GPU accrues cost continuously through financing, depreciation, power and cooling, whether or not it is doing anything useful.
• Most build-versus-buy models are run at an implied utilisation near 100%. Production rarely gets close, and the gap multiplies your effective cost per token.
• Idle GPUs are harder to redeploy than idle aircraft, because a free GPU can only absorb a job whose memory, latency and duration profile it can actually serve.
• High average utilisation on a dashboard can coexist with a queue of jobs waiting for a GPU shape that is busy.
• API prices are deflating while your hardware depreciates on a fixed schedule, which pushes the breakeven point further out every quarter.

Why is an idle GPU the right thing to worry about?

The aviation comparison is the cleanest way in. An aircraft's costs accrue by the calendar hour through financing, depreciation, insurance, maintenance and crew contracts. Its revenue accrues only by the flight hour. Every hour on the ground shrinks the revenue side while the cost side keeps running. Airlines with comparable fleets on comparable routes end up with very different economics, and most of the gap traces back to that one ratio.

A GPU works the same way. Financing, depreciation, power and cooling accrue continuously. Output accrues only when something useful is running. Two companies with the same GPU budget diverge on how much of that hardware is doing work at any given moment, not on how much they own.

What does the build-versus-buy spreadsheet usually get wrong?

It assumes the fleet flies.

The standard model compares a rising API line, where cost scales roughly linearly with tokens, against a flat owned-hardware line, and finds a crossing point. Past that point owning looks cheaper, so the conclusion is buy.

The flat line is almost always drawn at an implied utilisation nobody achieves. Divide the same fixed cost by 35% realised utilisation instead of 100% and your effective cost per GPU-hour roughly triples. The breakeven point moves far to the right, and for a lot of products it moves past the volume they will ever reach.

So before you model anything else, model the utilisation you will actually hit. Not the one you would hit if demand were flat.

Why do busy clusters still waste capacity?

Because GPUs run around the clock and demand does not. Infrastructure gets sized for the peak, the moment training runs, batch jobs and real-time traffic all land together, which leaves a meaningful share provisioned and unused the rest of the time.

Better forecasting would fix that if every GPU could absorb every kind of work equally well. It cannot. Real-time inference needs low latency, because a slow response counts as a failed one. Batch work wants throughput and tolerates hours of delay. Fine-tuning can hold a GPU for days. Quantisation needs a lot of capacity, briefly. A scheduler tuned for one of these misallocates the others by default.

Here is where the aircraft analogy breaks, and the break is the useful part. An idle 737 in Chicago can fly to Denver instead of Dallas. An idle GPU can only take a job whose memory, latency and duration profile it can serve. That is why a cluster can report high average occupancy while jobs sit queued waiting for a specific shape.

How should a founder decide, practically?

Start from three numbers rather than two.

Realised utilisation. Measure it over a full month including nights and weekends, not during a load test. If you have not run the cluster yet, model a pessimistic band and a realistic one, and see whether the decision flips between them. If it does, you do not have a decision yet.

Workload shape mix. Count how many distinct latency and duration profiles you run. One shape is schedulable by hand. Four shapes on one cluster is an orchestration project with an owner and a budget, and that cost belongs in the comparison.

The direction of the API price line. This is the observation the infrastructure framing tends to leave out. Your hardware depreciates on a fixed schedule that was set the day you signed. Provider prices are moving the other way, fast: a small tier repriced 80% downward this summer now matches what a flagship scored in March at roughly one-thirteenth the token price. A breakeven you calculate against today's API prices is a breakeven against a line that keeps dropping underneath you. Run the comparison against a deflating API curve, not a flat one, and be honest about how long your depreciation schedule has to hold up.

None of this argues against owning hardware. It argues that the sticker price per GPU-hour is not the number that decides it. Effective cost per useful compute hour is, and that is a function of utilisation you have to earn continuously rather than buy once.

Takeaway: the cheapest GPU is the one you are already running at high utilisation, and the most expensive one is the one you bought against a spreadsheet that assumed you would.

If you want to sanity-check the decision, you can model API cost against effective owned-hardware cost at several utilisation levels in Calcaas before you commit to a purchase.