Where randomness meets reason

Office Hours — How do I actually build production LLM systems when the cheaper models feel too risky and the expensive ones blow through my budget? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-10-22T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — How do I actually build production LLM systems when the cheaper models feel too risky and the expensive ones blow through my budget?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

office-hoursq-and-apractical-ai
Daily One question from the trenches, one opinionated answer.

How do I actually build production LLM systems when the cheaper models feel too risky and the expensive ones blow through my budget?

This is the trillion-dollar question right now, and the honest answer is that the “safe” choice (always use the flagship model) and the “cheap” choice (always use the budget tier) are both wrong at scale. You need a tiered strategy.

The Real Cost Problem

Cheaper models aren’t just slower—they’re systematically worse on your specific workload in ways that matter. Claude Haiku 5.5 costs 75% less than Opus 5.5, but it’s not 75% as good. On some tasks it’s fine; on others it fails in expensive ways. A routing layer that picks the wrong model burns tokens on retries, hallucinations, and cascading errors. I’ve watched teams save $50K on model costs and lose $200K to customer support and re-runs.

The cost of a hallucination isn’t just the API call. It’s the retry token cost, the human time to review it, the customer impact if it ships, or the compliance risk if it’s wrong.

The Three-Tier Approach

Build your system with three inference paths, not one.

Tier 1: Cheap/Fast (Claude Haiku 5.5, GPT-6 Luna, DeepSeek V4.1-Flash). These run everything by default. Haiku 5.5 has tiered pricing ($0.10/$0.50 per M for prompts up to 100K tokens, then $0.50/$2.50 above that), so your prompt size matters. Luna is a flat $0.10/$0.50. Use these for classification, straightforward extraction, retrieval ranking, or any task where you can measure success objectively (pass/fail test, schema validation, exact match).

Tier 2: Mid-range (Claude Sonnet 5.5, GPT-6.1 Sol). Run here only after a cheap model fails or when the task genuinely needs better reasoning. Sonnet 5.5 is $2/$10 (cached reads $0.10). This isn’t much more expensive than Haiku if your prompts are cached. Sol ($2/$10 per M, cached $0.10) is roughly equivalent and slightly better on coding/agentic tasks. Use this tier for tasks requiring multi-step reasoning or domain judgment.

Tier 3: Flagship (Claude Opus 5.5, GPT-6 Astra). Only for truly ambiguous or high-stakes decisions. Opus 5.5 is $4/$20 per M (cache reads $0.20). Astra is $10/$50 per M and genuinely overkill for most production workloads. Reserve it for rare escalations or tasks where failure has massive business impact.

Concrete Example: Document Classification

Say you’re classifying support tickets into 12 categories.

def classify_ticket(ticket_text: str):
    # Try cheap first
    result = claude_haiku.messages.create(
        model="claude-haiku-5-5",
        max_tokens=50,
        messages=[{
            "role": "user",
            "content": f"""Classify this ticket into one of: {CATEGORIES}
            
Ticket: {ticket_text}

Respond with ONLY the category name."""
        }]
    )
    
    category = result.content[0].text.strip()
    confidence = extract_confidence(result)  # Check for hesitation markers
    
    # Fall back to mid-tier if unsure
    if confidence < 0.7 or category not in CATEGORIES:
        result = claude_sonnet.messages.create(
            model="claude-sonnet-5-5",
            max_tokens=100,
            messages=[{
                "role": "user",
                "content": f"""You are a support ticket classifier. Think through this carefully.

Ticket: {ticket_text}

Which of these categories fits best? {CATEGORIES}

Explain your reasoning briefly, then state your final answer."""
            }]
        )
        category = parse_final_answer(result)
    
    return category

At scale, this runs 80-90% of tickets on Haiku. The 10-15% that need escalation go to Sonnet. You pay ~80% less than always using Sonnet, and you’re more reliable than always using Haiku.

Cache Everything Aggressively

Prompt caching is not optional—it’s your actual margin. If your system prompt, context, or retrieval results are reused, cache them.

system_prompt = """You are a technical support agent...
[long context about policies, product info, etc.]"""

response = client.messages.create(
    model="claude-haiku-5-5",
    max_tokens=500,
    system=[{
        "type": "text",
        "text": system_prompt,
        "cache_control": {"type": "ephemeral"}  # Cache for 5 min
    }],
    messages=[{
        "role": "user",
        "content": user_query
    }]
)

On Haiku, cached reads are $0.01 per M (compared to $0.10 for normal input). On Sonnet, they’re $0.10 (vs. $2 normal). For a system handling thousands of requests daily against the same context, caching cuts your bill in half or more.

Know Your Fallback Signal

The tiering only works if you have a reliable signal for “this failed.” Options:

  • Schema validation: Did the model return valid JSON matching your schema?
  • Objective test: Does the output pass a programmatic check (regex match, fact lookup, unit test)?
  • Confidence marker: Is the model hedging (“I’m not sure,” “This might,” “It’s unclear”)?
  • Retry counter: Are we hitting the same input repeatedly?

Don’t use subjective human judgment as your fallback signal in production. That scales linearly with load and defeats the point.

The Real Budget Control

Set per-request budgets, not per-month budgets.

def call_with_budget(task, max_input_cost_cents=1, max_output_cost_cents=5):
    # Tier 1: ~0.005 cents per 1K input tokens (Haiku)
    if estimated_input_tokens * 0.000005 < max_input_cost_cents:
        return try_haiku(task)
    
    # Tier 2: ~0.02 cents per 1K input (Sonnet)
    if estimated_input_tokens * 0.00002 < max_input_cost_cents:
        return try_sonnet(task)
    
    # Tier 3: ~0.1 cents per 1K (Opus/Astra)
    return try_opus(task)

This prevents a single request from exploding costs. If a task would cost $10 to solve with the right model, you either architect differently or accept the cost explicitly.

Common Mistakes

Teams often pick one model and stick with it because “evaluating models is hard.” They’re right that it’s hard, but not picking is more expensive. You need a 2-hour eval on your actual workload (50-100 real examples, check pass rate and cost) before shipping. That’s worth it.

Also avoid “backup” strategies where you call cheap, then call expensive only on failure. That doubles your latency and costs on the failure path. Better to predict upfront whether you’ll need help (based on input length, domain keywords, or a quick cheap classification step).

Bottom line: Build a tiered inference system that routes simple tasks to cheap models and escalates only what needs it. Implement this with objective success signals and aggressive caching. You’ll end up spending 40-60% less than always using midrange models while shipping more reliable results than always using budget tiers.