Office Hours — What are you using for LLM inference in production?

What are you using for LLM inference in production?

We’re running a polyglot stack because no single provider or model wins across all our workloads. Most teams claim they’re “API-only” until they hit latency walls, cost cliffs, or compliance constraints. Then things get real fast.

The Tier Strategy

For customer-facing features where latency matters, we’re on Gemini 3.5 Flash for agentic tasks (it’s just faster at reasoning-per-token than competitors) and Claude Opus 5 for high-stakes reasoning where accuracy beats speed. Both hit our SLA, though at wildly different costs.

For batch work and retrieval-augmented generation, we switched to Gemini 3.7 Flash after it dropped in August. The introductory pricing through Dec 31 made the math work, and the long context (1M tokens) lets us avoid chunking that was killing latency downstream. We’re building the cost model now for when pricing rises Jan 1.

For cost-sensitive summarization and routing, GPT-4.1 Nano ($0.10/$0.40 per M) is our workhorse. It’s cheap enough that we don’t sweat retries, and the 1M context window means we can process entire documents without splitting.

Where We Diverged From “Just Use Claude”

Early on we standardized on Claude Opus 4.8, and it was genuinely good. But when we benchmarked against our own codebase for a code-assist feature, Claude Sonnet 5 closed most of the gap while cutting costs by 60%. That forced a recount: if models are converging on quality, infrastructure and orchestration matter more than raw capability now.

The trap is believing your vendor’s benchmarks. We ran 50 tasks from our actual queue across four models and measured real wall-clock time, token consumption, and error rate. Claude Opus 5 won on coding, but Gemini 3.5 Flash won on agentic orchestration because it returned structured outputs faster. GPT-5.6 Sol was best for multi-hop reasoning but too expensive for our volume. One data point per model per use case beats ten published benchmarks.

The Infrastructure Reality

API calls are easy. Everything else is hard.

We built a router that maps request type to model, with fallback logic for quota exhaustion (Claude → Gemini → GPT). Without it, a spike on one provider cascades into latency everywhere. The code is maybe 200 lines, but the operational discipline is immense.

Caching is non-obvious. Both Anthropic and OpenAI offer prompt caching, and we use it for static context (our API docs, company style guide, code standards). But you need to benchmark: if your cached content changes every 12 hours, you’re wasting compute on cache misses. We measure cache hit rate per endpoint and adjust retention policies quarterly.

Token budgets are a lie if you don’t enforce them. We set hard caps per request ($0.50 on fast paths, $2 on slow paths) and let inference fail rather than exceed them. That sounds draconian until you see a runaway loop burn $4k before anyone notices. The cap forces you to design systems that work within constraints, not systems that hope constraints don’t matter.

Where Open Models Fit

We run Qwen 3.6-27B locally on two H100s for internal tools—documentation search, log analysis, dev assist. Not in production for customers, but real enough that we’re tracking inference time and accuracy. The appeal is dead simple: no API calls, no latency jitter, no rate limits, no token bills. The cost is ops overhead we’d rather not have.

For most teams, local open models are a distraction. If you’re not deploying to your own infrastructure, the operational burden exceeds the savings. We could justify it because we already run Kubernetes and had spare GPU capacity. If you’re API-only, stay API-only.

The Cost Reality

Last month we spent $18k on inference across a user base of ~50k. That’s about $0.36 per active user per month. It sounds fine until you realize two coding-assist requests per user per day is already pushing us toward $1.20/user at current models. The math doesn’t scale without deeper cost optimization.

We’re cutting context length where it doesn’t hurt quality, batching requests that don’t need real-time responses, and increasingly using cheaper models as routers to decide whether to call expensive models. An example:

# Route decision: is this request worth Claude Opus?
def should_use_expensive_model(query):
    # Use fast, cheap model to decide
    signal = gemini_flash.classify(
        query,
        categories=["needs_deep_reasoning", "routine_lookup", "junk"]
    )
    if signal == "needs_deep_reasoning":
        return claude_opus.answer(query)
    else:
        return gemini_flash.answer(query)

That shaved 35% off inference costs by avoiding Claude on requests it would waste capacity on anyway. The cost of the routing call is negligible.

Real Production Gotchas

Rate limits will surprise you. We hit OpenAI’s quota on a Tuesday at 2 PM and had no fallback. Now we batch non-urgent work and request quota increases weeks before we need them.

Model versions matter more than you think. Upgrading from Gemini 3.1 Flash to 3.5 Flash in one afternoon caused a silent latency increase across two services because the newer model was more conversational and returned longer outputs. We had to adjust parsing downstream. Test model changes in staging, and measure wall-clock time, not just accuracy.

Tool use is unreliable for production workflows. When we gave Claude access to an API via tool calling, it hallucinated valid-looking tool calls that didn’t match our schema. Now we use structured outputs (Responses API for GPT, JSON mode for Claude) and validate schemas before calling anything external.

Context window depth is a lie. Models work well at the start and end of long context but drift in the middle. If your task requires reasoning over document pages 45–55 in a 100-page file, expect degradation. We started chunking at 100k tokens even though 1M is available.

Bottom line:

Use frontier models (Claude Opus 5, Gemini 3.5 Flash, GPT-5.6 Sol) only for irreplaceable reasoning tasks, route everything else through cheaper tiers (GPT-4.1 Nano, Gemini 3.7 Flash, Claude Sonnet 5), and measure costs and latency on your actual workload—not benchmarks. The savings are in architecture and orchestration, not model choice.

Question via Hacker News