Where randomness meets reason

Office Hours — How do I decide if I need a fancier model, cheaper model, or just a better prompt for my specific task? A daily developer question about AI/LLMs, answered with a direct, opinionated take. 2026-10-23T12:00:00.000Z Office Hours Office Hours office-hoursq-and-apractical-ai

Office Hours — How do I decide if I need a fancier model, cheaper model, or just a better prompt for my specific task?

A daily developer question about AI/LLMs, answered with a direct, opinionated take.

office-hoursq-and-apractical-ai
Daily One question from the trenches, one opinionated answer.

How do I decide if I need a fancier model, cheaper model, or just a better prompt for my specific task?

This is the question that separates people debugging in production from people throwing money at the problem. And the answer is: you need a way to measure what actually matters on your data, not leaderboards.

The measurement trap

Everyone starts here: “Claude Opus is better, so we use Opus.” Or: “GPT-6 Luna is cheaper, let me swap it in.” Then production breaks or the bill surprises you. The issue is that leaderboards measure averaged performance across thousands of tasks. Your task is one specific thing. A model that ranks #2 overall might be #1 for your exact use case.

You need a small eval set from your actual workload, scored on metrics that align with what breaks in production. Not accuracy—that’s useless. You care about: does it miss critical details (false negatives), does it hallucinate fields that don’t exist (false positives), does it format output in a way your parser can consume reliably, does it take so long that users click away.

Model vs. prompt vs. luck

Once you have evals, run a simple experiment. Take your current setup. Then:

  1. Keep the model the same, improve the prompt. Spend two hours on example selection, instruction clarity, output format specification. Re-run evals.
  2. Keep the prompt the same, downgrade to a cheaper model (or upgrade to a fancier one). Re-run evals.
  3. Try a completely different model tier at the same prompt.

What you’ll often find: a crisp, specific prompt on Claude Sonnet 5.5 ($2/$10 per M) beats a vague prompt on GPT-6 Astra ($10/$50 per M). Not because Sonnet is better at your task—because your prompt was bad. The expensive model just mask-fitted the bad prompt.

A concrete example

A support ticket classification task. You think: “This needs GPT-6 Astra reasoning because ticket text is messy.” You benchmark it:

eval_set = 50 real tickets with ground-truth categories
categories = ["billing", "technical", "account", "other"]

# Version 1: Claude Sonnet 5.5, generic prompt
prompt = "Classify this ticket: {ticket}"
accuracy = 82%, false_positive_rate = 6%

# Version 2: Same model, few-shot + structured output
prompt = """Classify this support ticket into one of: billing, technical, account, other.
Show your reasoning. Respond in JSON.

Example 1: "My card was declined"
-> {"category": "billing", "reasoning": "payment error"}

Example 2: "App keeps crashing when I upload photos"
-> {"category": "technical", "reasoning": "crash on specific action"}

Ticket: {ticket}
"""
accuracy = 91%, false_positive_rate = 2%

# Version 3: Swap to GPT-6 Luna ($0.10/$0.50), keep Version 2's prompt
accuracy = 88%, false_positive_rate = 3%

Version 2 outperformed Version 3 (91% vs. 88%), and costs 40x less. You never needed Astra.

When to actually upgrade the model

Upgrade when:

  • You’ve squeezed the prompt: few-shot examples added, instructions clarified, output format locked down.
  • Your eval shows the model is hitting a systematic failure mode. For example, it consistently misses edge cases, or it hallucinates in predictable ways. That’s a model limitation, not a prompt problem.
  • The cost-quality math makes sense. If upgrading from Sonnet to Opus 5.5 ($2/$10 vs. $4/$20) improves accuracy from 90% to 95%, and you’re running this 10,000 times/day, you might pay $86K/month extra for 5% better accuracy. Is that ROI positive? Only you know.

When prompt engineering hits the wall

You’ll know you’re there when:

  • Adding more examples doesn’t help. The model is hitting the ceiling of what it can do with the task structure you’ve given it.
  • You try 5+ different prompt framings and all plateau at ~90% accuracy. The last 10% requires the model to genuinely understand something it doesn’t.
  • Chain-of-thought, step-by-step, persona-based prompts all fail equivalently. The model doesn’t benefit from reasoning scaffolding because it fundamentally doesn’t see the pattern you need.

At that point, you have two real options: upgrade the model, or change the task (add preprocessing, break it into subtasks, use a hybrid approach with a cheaper model for triage and an expensive one for hard cases).

The sneaky cost trap

Model routing looks cheap in theory. “Route easy tickets to Luna, hard ones to Opus.” But there’s overhead: you need a classifier to decide which is which. That classifier itself can be wrong. And every time you misroute, you either retry (token waste) or produce garbage (quality loss). A routing classifier error costs more than just running everything on a consistent model. Databricks found this: they tried routing OpenAI models by complexity, then realized they were burning tokens on the routing model and error recovery. They ended up standardizing on a single mid-tier model for most workloads.

Bottom line:

Build a small eval set on your actual data, measure prompt improvements first (usually 2-4 hours of iteration), then A/B model swaps. Most people overspend on models because they never baseline their prompt quality. The cheapest optimization is a clearer prompt.