Office Hours — How do you handle cost and latency tradeoffs when choosing between different LLM providers and model sizes for your application?

How do you handle cost and latency tradeoffs when choosing between different LLM providers and model sizes for your application?

This is the decision that separates hobby projects from production systems. You have real constraints—you need a response in under 200ms for a user-facing feature, or you need to process a million documents a day without burning the budget. The model tier you pick, the provider you choose, and the serving infrastructure you build around it all collide in ways that aren’t obvious until you’re already in production.

Cost isn’t just per-token; it’s per-task

Pricing pages show $/1M tokens, but that number is meaningless without knowing how many tokens your actual workload burns. A task that looks cheap at $0.0001/token can become expensive if it requires five retries, or if your agent loops internally while reasoning through a problem.

Hidden token costs in agentic systems are brutal. You might call GPT-5.6 Sol at $0.80/M input tokens, think you’ve got a cheap pipeline, then discover your agent’s internal reasoning loop, the clarifying re-prompts, and the retry logic are actually consuming 5x the tokens of the “happy path.” ML Mastery has already documented this: agentic systems burn through budget in ways that aren’t obvious from a single API call. Build token accounting into your observability from day one. Log every API call with input/output token counts, then aggregate by task, by hour, by customer, by failure reason. If you don’t know where the tokens are going, you don’t have a cost problem yet—you have a visibility problem.

Model selection is not about capability alone; it’s about your actual workload

The current frontier is crowded. Claude Opus 5 matches Fable 5–level performance on coding and knowledge tasks at 50% lower token cost. GPT-5.6 Sol has been cut down to an 80% price reduction in four months through distillation. Gemini 3.5 Flash leads on agentic and coding benchmarks with ~4x faster output than other frontier models. Qwen3.8-Max ranks second on Vision Arena and is strong on long-horizon tasks. They’re all competitive on benchmarks, but they behave completely differently on your specific problem.

Databricks benchmarked coding agents on their own million-line production codebase and found that open-source GLM-5.2 matched Claude Opus 4.8 while cutting costs from $1.94 to $1.28 per task. They now run GLM-5.2 as default. Vendor benchmarks are gaming chairs—your codebase is your real benchmark. Build a small eval on your actual workload (20-50 real examples) and test three models: the frontier option, a mid-tier option, and an open-source contender if your use case supports it. Measure both quality and tokens. The winning model might surprise you.

Latency and throughput have different solutions

If you need low latency for a single user request, you’re optimizing for inference speed. Gemini 3.5 Flash is legitimately ~4x faster output than competitors—if you can tolerate its quality trade-offs (usually acceptable for coding and agentic work), speed wins. GPT-5.6 Luna tier is designed for low-latency tasks. For high-throughput workloads (batch processing millions of documents), you optimize differently: continuous batching strategies on self-hosted infrastructure or batch APIs can be 3-5x cheaper than per-request inference, even at the cost of higher wall-clock latency.

The architectural choice matters more than the model. If you’re serving individual API requests through a standard inference endpoint, you pay per-request overhead. If you can batch (even loosely, with a 5-10 second batching window), you shift to a throughput-optimized regime where infrastructure costs drop significantly. Baseten published deep work on this: static vs. dynamic vs. continuous batching for LLM inference have radically different latency-throughput tradeoffs, and getting it wrong leaves 3-5x performance on the table.

A concrete example: the cost-latency grid

Let’s say you’re building a coding copilot that needs to give feedback on a pull request. One approach:

Option A: Frontier model, per-request
- Use Claude Opus 5 for every request
- 50KB average input (code + context)
- Latency requirement: 5 seconds
- Cost per request: ~$0.15
- Daily volume: 1,000 requests
- Daily cost: $150

Option B: Tiered model with fast-track fallback
- Use Claude Sonnet 5 first (~95% of requests complete in 2 seconds)
- Fall back to Opus 5 only if Sonnet output fails a deterministic check (syntax error, no suggestion)
- Cost per request: ~$0.04 (mostly Sonnet) + occasional Opus override
- Daily volume: 1,000 requests
- Daily cost: ~$55 (estimated, assuming 5-10% fallback rate)
- Latency: 2 seconds for 90% of requests, 5 seconds for 10%

Option C: Async batch with local fast model
- Use open-source DeepSeek V4-Flash 0731 for initial analysis (instant, local)
- Batch requests to Claude Sonnet 5 for refinement (runs every 30 seconds in batches of 20)
- Users see local results in 100ms, refined results arrive 30-60 seconds later
- Cost per request: ~$0.02 (mostly inference cost, batching amortizes the API call)
- Daily volume: 1,000 requests
- Daily cost: ~$20
- Latency: 100ms to first response, 30-60 seconds to final refined response

Option B saves 63% vs. Option A. Option C saves 87% but requires accepting async UX. The right choice depends on your users’ tolerance for latency and how much you care about initial response time vs. final quality.

Build for optionality early

Avoid hard-coding a single model. Parameterize model selection by task type or user tier. Use an abstraction layer (like LangChain’s LLMChain or instructor for structured output) so swapping models doesn’t require rewriting your prompts. You’ll want to A/B test later, and switching from Claude to GPT-5.6 Sol to Gemini should be a config change, not a refactor.

Claude Sonnet 5 has a new tokenizer that emits ~30% more tokens than the old version. This means real per-task costs are roughly 40% higher than the list price suggests—a detail you only catch if you’re actually logging token usage. Logging token counts is non-negotiable. You need to see when a model suddenly burns 2x more tokens on the same workload (could signal a tokenizer change, or it could signal your prompt is drifting).

Batch APIs for throughput

If you can accept 1-24 hour latency, batch APIs cut costs dramatically. OpenAI, Anthropic, and others offer batch endpoints that are 50% cheaper than per-request inference because they optimize packing and scheduling. Document processing, log analysis, or overnight report generation are obvious candidates. The tradeoff is latency, but if you don’t need real-time results, this is free money.

Monitor quality separately from cost

Cost optimization is a trap if it means your model starts hallucinating or producing garbage. Build a small automated eval that runs hourly on a sample of production outputs. Check for red flags: is the model confident but wrong more often? Is it refusing requests it used to handle? Is token count creeping up? These are signals that your cost-saving model swap degraded quality in ways that metrics don’t catch. A 40% cost reduction that increases support tickets by 30% is a net loss.

Bottom line: Build an eval on your actual workload, not benchmark scores. Measure tokens per task and latency together, then test a tiered approach (fast cheap model for 80% of requests, frontier model for the hard 20%) before assuming you need one model for everything. Log token usage obsessively—it’s your most reliable signal for cost creep and model drift.

Question via Hacker News