Office Hours — How do I route requests to the cheapest model without tanking quality, and what am I missing about the real costs?
A daily developer question about AI/LLMs, answered with a direct, opinionated take.
How do I route requests to the cheapest model without tanking quality, and what am I missing about the real costs?
The instinct is right: cheaper models can handle a lot of tasks that don’t need frontier capability. The mistake is thinking the cost calculation stops at per-token pricing.
The Math You See vs. The Math That Matters
OpenAI’s GPT-6 Luna costs $0.10/$0.50 per million tokens. Claude Haiku 5.5 under 100K tokens runs $0.10/$0.50. DeepSeek’s off-peak rates are half that. On paper, route everything to Luna or Haiku and pocket the savings.
In practice, switching models kills the margin you thought you were saving.
A developer ran three office automation agents (Lindy, Manus, Claude Free) 27 times each across real tasks—reconciliation, data entry, research. The median cost-per-task ranged from $2 to $8, but the cheaper systems failed more often, required retries, and sometimes needed human cleanup that nullified the savings entirely. One task that succeeded at $4 with Claude Free cost $12 total when you added the operator’s time to verify and correct the output.
Where Routing Actually Costs You
The hidden tax shows up in three places:
First, context window mismatches. GPT-6 Luna has 128K context. If your task needs to fit a legacy codebase, a full conversation history, and retrieval results, you either truncate (quality drops) or re-retrieve context on every call (tokens spike). A cheaper model that can’t hold context often costs more in retries and re-prompting than a pricier model that nailed it the first time.
Second, failure and retry cascades. When a cheaper model hallucinates a tool call or invents a field, your orchestration layer retries. That’s another full inference call, often with more tokens because you’re adding error context or a corrected prompt. Databricks benchmarked their million-line codebase and found GLM-5.2 matched Claude Opus 4.8’s task success rate but at lower per-task cost—the key word is matched success rate. A model that fails 15% of the time doesn’t save money; it burns it on retries.
Third, model-specific prompt inflation. GPT-6 Luna sometimes needs more explicit instruction than Opus to stay on track. You end up padding the prompt, which inflates input tokens and erases the per-token savings. A Haiku 5.5 prompt that’s 8K tokens versus an Opus prompt that’s 5K still looks cheaper until you multiply by volume.
A Real Routing Strategy That Works
Don’t route by default. Route by task category with explicit success criteria.
For tasks where success is verifiable and fast (classification, extraction, tool calling with immediate feedback), cheaper models often win. A support ticket classifier or API call validator can run on Luna or Haiku with simple test-first validation. If it passes your eval on a small batch, roll it out. If it drifts, you’ll see it in the metrics.
For ambiguous or long-horizon tasks (code review, architectural decisions, research synthesis), stick with a capable model until you have real data showing a cheaper alternative succeeds at the same rate. “Ambiguous” means you can’t write a test that catches failure—you need human judgment or downstream metrics to know if it worked.
Token caching flips the equation. Claude’s cache reads cost $0.20/M for Opus, $0.10/M for Sonnet, versus full input prices of $4/$2. If you’re calling the same model repeatedly on similar tasks (same documents, same instructions, different queries), caching saves more than model switching. Gemini 4 Argon’s cached input runs at 95% off—if your workload benefits from caching, switching models to save 30% on input cost looks silly when the better move is caching 95% of it.
What You Should Actually Measure
Skip per-token cost. Measure cost-per-successful-task, including retries and cleanup.
If routing splits traffic between two models, track success rate, retry rate, and human review rate separately. A task that costs $0.50 with Luna but fails 20% of the time is more expensive than a task that costs $1.50 with Opus and succeeds 95% of the time. The breakeven point depends on your retry cost, not on advertised pricing.
One more thing: time matters. If Luna takes 3 seconds and Opus takes 1 second, and your application can’t queue requests, the latency cost of waiting compounds across thousands of calls. Frontier models are faster. That’s worth money in user-facing systems.
Bottom line: Route by task category and success metrics, not by headline pricing. Model switching usually costs more than the savings it promises unless you’ve validated that the cheaper model actually succeeds at the same rate. Use caching and task-specific batching to compress costs before you even think about model selection.