Self-Hosting vs API: When Local LLMs Actually Cost Less

Key takeaways

• Free local inference usually means no per-token API bill, not zero cost.
• Reusing existing hardware (a dev machine, an idle server) is what makes local models cheap.
• Per-token APIs win for low or bursty volume; local wins for steady, high-volume background jobs.
• Quality and latency trade-offs decide whether a smaller local model is good enough.
• Compare local cost per task against the API cost per task before committing.

What does free local inference really mean?

It means you stopped paying per token to a provider, not that the work costs nothing. When a team runs an open model locally to handle a background task like triaging pull requests or issues, the marginal cost of each run drops toward the cost of electricity and the time the machine is busy. If the hardware was already paid for and otherwise idle, that can round to almost nothing.

The honest accounting adds three buckets the API hides: hardware (or its amortized share), electricity and cooling, and the engineering time to set up and maintain the stack. Free is the marginal view; cheap is the full view.

When do local models genuinely beat an API?

For steady, high-volume, latency-tolerant work. Background automation is the sweet spot: tasks that run constantly, do not need the strongest frontier model, and can wait a second or two. Repo triage, classification, tagging, and bulk enrichment all fit. Here local inference on owned hardware can undercut per-token API pricing by a wide margin because you are not paying a premium per call.

When does a per-token API still win?

For low, bursty, or quality-critical work. If volume is small, the API bill is trivial and not worth replacing with hardware and maintenance. If load is spiky, you would buy hardware that sits idle most of the time, which is the opposite of efficient. And if the task needs top-tier reasoning, a smaller local model may not clear the quality bar, and a cheaper-looking option that produces worse results is not actually cheaper.

How do you decide? A quick founder playbook.

Estimate cost per task both ways. For the API, that is tokens per task times the provider's rate. For local, it is the machine's running cost for the time the task takes, plus an amortized slice of hardware and setup. Then multiply by tasks per month.

For example, say a triage task is cheap per call on an API but you run tens of thousands a month. The API total adds up, while the same job on an idle machine you already own costs little beyond power. Flip the volume to a few hundred a month and the API almost always wins on simplicity. Treat these as illustrative and plug in your own numbers.

The takeaway: local models are not free, they are cheap at high, steady volume on hardware you already have. Model local cost per task against API cost per task in Calcaas before you migrate a workload.