Office Hours — How do I decide between using smaller, faster language models versus larger ones for latency-sensitive applications?

How do I decide between using smaller, faster language models versus larger ones for latency-sensitive applications?

It depends on whether you can afford to trade accuracy for speed, and whether your latency constraint is hard or soft. Start by measuring: run both a small and large model on your actual task, measure p95 latency and error rate, then calculate the cost of latency failures in your domain. If you’re serving real-time search results or financial trading signals, latency failures might cost more than accuracy improvements. If you’re batch processing internal reports, small accuracy gains might justify slower inference.

The tradeoff is not straightforward

Smaller models are faster per token, but they often need longer prompts to work reliably. A 13B model might need you to spell out every step of reasoning explicitly, while a frontier model grasps context faster. That means your effective latency is often (tokens_generated / model_throughput) + (extra_tokens_needed_for_clarity / speed_multiplier). You can’t just divide token count by throughput and call it done.

Claude Haiku 5.5 pushes $0.10 input / $0.50 output per million tokens on prompts up to 100K, making it dirt cheap for high-volume latency-tolerant work. But if you’re building a customer-facing feature where 500ms feels like lag, you need to test whether Haiku’s output quality lets you keep prompts short enough to stay under your latency budget. Sometimes the answer is no.

Concrete example: a coding assistant feature

Say you’re adding code completion to an IDE and need a response in under 300ms. GPT-6 Astra ($10/$50 per M) will give you better completions but takes 800ms at peak load. Gemini 3.8 Flash ($0.75/$3.75 through Dec 31) gives you 150ms latency and handles 80% of cases correctly. Your move: use Gemini 3.8 Flash for the hot path, fall back to Astra only when the model flags uncertainty. This hybrid approach costs you maybe 5% in total inference spend but keeps 95% of requests fast.

import anthropic
import time

def fast_code_complete(code_prefix: str, use_frontier: bool = False) -> str:
    client = anthropic.Anthropic()
    
    if use_frontier:
        model = "claude-opus-5-5"  # Frontier accuracy, slower
    else:
        model = "claude-haiku-5-5"  # Fast, good enough for most completions
    
    start = time.time()
    response = client.messages.create(
        model=model,
        max_tokens=150,
        messages=[{
            "role": "user",
            "content": f"Complete this code:\n{code_prefix}"
        }]
    )
    latency_ms = (time.time() - start) * 1000
    
    # If latency exceeded budget and we used the fast model, 
    # you might retry with caching or different prompt framing
    if latency_ms > 300 and model == "claude-haiku-5-5":
        print(f"Fast model hit latency at {latency_ms}ms—consider prompt optimization")
    
    return response.content[0].text

The question isn’t really “small vs. large”—it’s “am I willing to pay for latency?” If you are, a frontier model wins. If you aren’t, you need to engineer the small model path: shorter prompts, better examples, constraint-based validation to catch failures before they reach the user.

Latency coupling and hidden costs

Smaller models running on CPU inference (Modal, RunwayML, or your own hardware) can hit lower latencies than a frontier model on shared API infrastructure, but only if you optimize the entire pipeline. Network overhead often dominates token generation time. A 50ms inference latency doesn’t matter if your network call takes 200ms. Measure end-to-end, not just model throughput.

Also: cached context is a secret latency killer. GPT-6 Luna ($0.10/$0.50) caches input at $0.01 per M tokens, making repeat queries on the same document nearly free. Claude Haiku 5.5 caches at $0.01 per M too. If your use case involves lots of repeated context (same user, same document, same conversation thread), caching can make a nominally slow model faster in practice than a small model without caching overhead.

When the small model fails

Latency failures often compound. If your small model hallucinates a tool call or produces malformed output, recovery is expensive: you retry with the frontier model, paying double latency and double cost. For tasks where correctness is verifiable (math, code, structured extraction), this matters. For tasks where output quality is subjective (creative writing, summarization), a small model’s consistent-but-mediocre output might beat a frontier model’s slow-but-brilliant one.

Test the failure path explicitly. Build a validator that catches when the small model confidence is low, then measure how often you actually need to escalate to the frontier model in production. If it’s more than 10%, the cost math changes.

The practical decision tree

Start small and measure. Deploy Claude Haiku 5.5 or Gemini 3.8 Flash to production behind a feature flag. Instrument latency and error rates on real traffic. If p95 latency < your budget and error rate < your tolerance, ship it. If latency is under budget but errors are high, add a fallback or improve prompts before upgrading models. Only reach for Opus or Astra if you’ve exhausted cheaper optimizations.

Don’t optimize for “best model” globally. Optimize for latency budget per feature. Different features have different tolerance: a search ranking endpoint might need 50ms, while a background document processor can burn 5 seconds. Use Claude Haiku 5.5 for the former, Opus 5.5 for the latter. Save frontier models for where they actually unlock capability you can’t get elsewhere.

Bottom line: Latency-sensitive applications rarely need the largest models. Test small models on your actual workload first, measure error rates and latency in production, and only upgrade if you’ve exhausted optimization (shorter prompts, better examples, caching, validation). The speed gain from a smaller model usually beats the accuracy gain from a larger one unless you’re hitting real failures.

Question via Hacker News