Office Hours — What frameworks or tools help you systematically evaluate whether an LLM is ready to use for a specific coding task?

What frameworks or tools help you systematically evaluate whether an LLM is ready to use for a specific coding task?

The honest answer is that most teams are still doing this wrong. You can’t rely on vendor benchmarks anymore. SWE-Bench Pro itself has ~30% broken tasks. Leaderboards are noise when the top two models show no statistically significant difference at conventional p-values. So you need to build your own signal.

Start with your actual workload, not published benchmarks.

Pull 20-50 real examples of the coding task you care about. Not toy problems. Not curated examples. Actual messy extracts from your codebase, your API docs, your bug tracker, or your production logs. If you’re using an LLM to write unit tests, grab real test files. If you’re using it to refactor legacy code, grab actual legacy code that your team has struggled with.

Then run each model candidate against those examples and measure what actually matters to you: did the code compile? Did tests pass? Would a senior engineer accept this in code review without rework, or would it require multiple rounds of fixes? How many hallucinated imports or undefined variables did it generate? What was the token cost per task?

Databricks did this on their million-line production codebase. They benchmarked GLM-5.2 against Claude Opus 4.8 and found GLM-5.2 matched Opus while cutting costs from $1.94 to $1.28 per task. That would’ve been invisible in published benchmarks. Vendor evals are gamed. Your evals are honest.

Measure what your team actually values.

One team might care most about latency. Another needs cost efficiency. A third needs safety—never running untested code without human review. These priorities completely change which model is “ready.”

Build a simple scorecard:

• Correctness: What percentage of outputs required zero human fixes? What percentage required one round? More than one?
• Token efficiency: What’s the average input and output token cost per task? Does caching help, or does your task have too much variability?
• Latency: If you need responses in under two seconds, which models can you actually use?
• Safety: Are hallucinated functions being generated? Are there imports from fake libraries? How often does it confidently suggest breaking changes to your codebase?
• Debuggability: When it fails, can you understand why? Are errors clear or buried in confabulated explanations?

Weight these based on your constraints. If you’re generating code that runs in a sandbox with automated testing, correctness matters most and you can tolerate slower models. If you’re doing real-time code completion in an IDE, latency is the gate.

Use a structured testing approach, not vibes.

Set up a simple eval harness. You can use tools like Promptfoo or build your own in Python (it’s not that hard). For each test case, capture:

• The input (code snippet, feature request, error message, whatever your task is)
• The expected output or acceptance criteria
• The model’s actual output
• Whether it passed

Run this against at least three models. Most people test against one and assume they’ve learned something. You haven’t. You need comparative signal.

Here’s a minimal Python example:

import anthropic
import openai

def eval_coding_task(model, task_input, expected_criteria):
    if "claude" in model:
        client = anthropic.Anthropic()
        response = client.messages.create(
            model=model,
            max_tokens=2048,
            messages=[{"role": "user", "content": task_input}]
        )
        output = response.content[0].text
    else:
        client = openai.OpenAI()
        response = client.chat.completions.create(
            model=model,
            max_tokens=2048,
            messages=[{"role": "user", "content": task_input}]
        )
        output = response.choices[0].message.content
    
    # Your custom pass/fail logic
    passed = check_output(output, expected_criteria)
    tokens_used = response.usage.prompt_tokens + response.usage.completion_tokens
    
    return {"passed": passed, "tokens": tokens_used, "output": output}

# Run against your test set
results = {}
for model in ["gpt-6-astra", "claude-opus-5-5", "gemini-4-argon"]:
    pass_rate = sum(eval_coding_task(model, t["input"], t["criteria"])["passed"] 
                    for t in test_cases) / len(test_cases)
    results[model] = pass_rate

The check_output function is where the work lives. For generated code, that might mean:

• Try to parse it as valid Python
• Run a linter against it
• Execute it against a test suite
• Check for hallucinated library calls
• Have a human reviewer score it on a scale (this is acceptable, this needs revision, this is trash)

Account for the real cost of integration.

A model might be “ready” on accuracy but create operational friction. Consider:

• Context window: If your task requires your entire codebase as context and the model can’t fit it, you’re now building a context-selection system. That’s work.
• Cache behavior: Do you have repeated prefixes in your prompts? Prompt caching on Claude Opus 5.5 or Sonnet 5.5 can cut input token costs by 90%. But only if your task structure supports it. If every request is unique, caching is useless.
• Rate limits: If you’re processing 10,000 coding tasks a day, can the API handle it? What’s the cost at that scale?
• Tool calling consistency: If your agent needs to call specific functions, which models actually call them reliably versus hallucinating? (Spoiler: cheaper models hallucinate tool calls more often.)

Know when “ready” means something different.

A model might be ready for:

• Interactive coding assistance (user sees output, can iterate). Latency and UX matter most. Cost matters less.
• Batch code generation (human reviews later). Correctness and cost matter. Latency doesn’t.
• Autonomous agents (no human in the loop between steps). Safety and reliability dominate everything else. You probably need explicit guardrails, output validation, and human approval workflows.

These have completely different readiness criteria. Claude Code is ready for autonomous coding because it has safety classifiers catching 89% of dangerous commands. GPT-6 Luna running fully unsupervised might not be, even if it’s smarter, because the safety story is different.

One practical signal: cost-per-fix ratio.

Take a model that passes 80% of your evals. How many tokens does it use per task? How long does a human take to review and fix the 20% that fail? If a human engineer spends 10 minutes fixing bad outputs and the model costs $0.50 per task, that human time probably costs you $3–5. At some volume, it’s cheaper to just use a better model, even if it costs 2x more per token.

Conversely, if your task is simple enough that Claude Opus is overkill and GPT-6 Luna works 95% of the time at a tenth the cost, paying for Opus is waste.

Bottom line: Build a 20-50 example eval on your actual workload, test at least three models against it, measure correctness and cost together, and weight your scoring based on what your team actually needs. Vendor benchmarks will lie to you. Your own data won’t.

Question via Hacker News