Good Enough and 100x Cheaper Is Beating Best-in-Class: The Economics of AI Simulation

Key takeaways

• Chasing the highest-capability model on every task is often the wrong optimization once you price in cost and iteration speed.
• A 10% quality gap can be irrelevant for many product surfaces, while a 100x cost gap is never irrelevant.
• The right question isn't which model is smartest, it's which model is good enough for this specific task, at what cost.
• Teams that route tasks by difficulty, cheap model first, escalate only when needed, capture most of the savings without a visible quality drop.

What's the actual claim?

The framing making the rounds: simulation and RL-driven approaches can deliver roughly 10% worse output quality than the frontier alternative, for around 100x lower cost and up to 10,000x faster iteration. Whether the exact multipliers hold for every task is beside the point. The shape of the tradeoff is the important part: a small capability gap can buy an enormous cost and speed advantage.

That's an uncomfortable idea for teams that have quietly settled into "always use the best model" as a default. It's an easy default to justify (nobody gets blamed for picking the smartest model), but it's rarely the cheapest way to build a good product.

Why "10% worse" rarely shows up where users notice

Model quality gaps are not evenly distributed across use cases. A 10% drop in benchmark performance might mean occasional awkward phrasing in a summary, a slightly less elegant refactor, or a classification that needs a light second check. For a huge share of production traffic, tasks like tagging, extraction, routing, first-draft generation, that gap is invisible to the end user but very visible on the invoice if you ignore it.

The tasks where capability actually matters (complex reasoning, high-stakes decisions, anything customer-facing where errors are expensive) are usually a minority of total volume, even if they get the most attention internally.

The real lesson: stop optimizing for capability alone

The uncomfortable version of this: if you're running every request through your most capable (and most expensive) model by default, you are almost certainly overpaying for a large share of your traffic. The fix isn't switch everything to the cheap model, it's building a routing decision into your product: cheap and fast by default, escalate to the frontier model only when the task actually calls for it.

This is a harder engineering problem than picking one model and moving on, but the economics reward it heavily. A 100x cost difference on 70% of your volume dwarfs whatever margin advantage you'd get from micro-optimizing prompt length on the expensive model.

How to apply this without guessing

Three steps that don't require trusting anyone's benchmark:

1. 1Bucket your traffic by task type and tag each with a rough "does capability actually matter here" label.
2. 2Run a sample of your lower-stakes buckets through a materially cheaper model and measure real output quality, not benchmark scores, against your current model.
3. 3Only keep the expensive model on buckets where the quality gap is visible to users or costly when wrong.

One line to remember: 10% worse is a rounding error next to 100x cheaper, if the 10% doesn't show up where your users are looking. Calcaas's leaderboard maps model capability against cost side by side, so you can see exactly where a cheaper model's capability-per-dollar actually lands for the kind of task you're routing.