Office Hours — How should you structure LLM evaluation and testing in your development pipeline to catch issues before production?

How should you structure LLM evaluation and testing in your development pipeline to catch issues before production?

The bad news: there’s no standard playbook yet. The good news: you don’t need one. Most teams are overthinking this. You need three layers, not twelve, and they map to what actually breaks in production.

Start with a baseline eval set, not a research benchmark

Build a small set of 20–50 real examples from your actual use case. Not synthetic. Not MMLU. Real inputs your system will see. For a customer support chatbot, that’s actual tickets. For RAG over PDFs, that’s queries your users actually ask. Version control these like you version code—they’re your contract with your model.

Run your baseline set against your current model and measure: accuracy (correctness), latency (speed), and cost (tokens burned). This becomes your smoke test. Every time you swap a model or tweak a prompt, you run this baseline and compare. You’re not trying to get a 99% score. You’re looking for regressions.

Here’s a minimal eval setup in Python using Claude:

import json
from anthropic import Anthropic

client = Anthropic()

test_cases = [
    {"input": "How do I reset my password?", "expected_contains": "account settings"},
    {"input": "Can you help me with billing?", "expected_contains": "support team"},
    # ... 18-50 more cases from your real traffic
]

def run_baseline(model_id, test_cases):
    results = []
    for case in test_cases:
        response = client.messages.create(
            model=model_id,
            max_tokens=500,
            messages=[{"role": "user", "content": case["input"]}]
        )
        output = response.content[0].text
        passed = case["expected_contains"].lower() in output.lower()
        results.append({
            "input": case["input"],
            "output": output,
            "passed": passed,
            "tokens": response.usage.output_tokens
        })
    return results

# Run against your current production model
baseline = run_baseline("claude-opus-4.8", test_cases)
passed = sum(1 for r in baseline if r["passed"])
print(f"Passed: {passed}/{len(test_cases)}")
print(f"Avg output tokens: {sum(r['tokens'] for r in baseline) / len(baseline):.0f}")

That’s it. You now have a regression detector. Run it before every deploy.

Layer 2: Catch the failure modes specific to your system

Don’t test “does the model follow instructions”—test “does the model do the specific thing your system requires.” This is where most evals fail. Teams test generic capabilities and miss domain-specific failure modes.

For a coding agent, that’s: does it actually run the code it writes? Does it check test output? Does it open a PR or just print results? For a RAG system, that’s: does it use the retrieved docs, or does it ignore them and hallucinate? Does it say “I don’t know” when the docs don’t contain the answer?

Build adversarial cases that target these. Add 5–10 cases that should fail or behave differently:

adversarial_cases = [
    {
        "input": "What's our Q3 revenue?",
        "context": "We only have Q1 and Q2 data in the docs.",
        "expected_behavior": "say_unsure",  # not hallucinate
    },
    {
        "input": "Refactor this function to use async/await",
        "code": "def slow_fetch(): requests.get(url)",
        "expected_behavior": "produces_syntactically_valid_code",
    },
]

This is where you catch the model being confident but wrong. Run these alongside your baseline. A 10% failure rate on adversarial cases is a red flag.

Layer 3: Cost and latency gates in your pipeline

Before shipping a new model version, measure tokens and latency on your baseline set. Set hard limits. If Gemini 3.5 Flash costs 40% less than Claude Opus 4.8 on your workload but outputs are slower by 300ms, that’s a tradeoff you need to make consciously, not discover in production.

def compare_models(baseline_cases, models):
    for model in models:
        results = run_baseline(model, baseline_cases)
        accuracy = sum(1 for r in results if r["passed"]) / len(results)
        avg_tokens = sum(r["tokens"] for r in results) / len(results)
        print(f"{model}: {accuracy*100:.0f}% accuracy, {avg_tokens:.0f} tokens")

compare_models(test_cases, ["claude-opus-4.8", "claude-sonnet-5", "gpt-5.5"])

Don’t do this: automated grading with LLMs

Tempting idea. You use an LLM to grade whether another LLM’s output is good. This almost always fails. The grading model becomes a bottleneck, doesn’t actually measure what matters, and introduces another source of hallucination. If you absolutely need to grade subjective outputs (like prose quality), use real humans on a sample—10–20 cases, not all 50. Faster, cheaper, more reliable.

Monitor what actually matters in production

Once you’re live, instrument the stuff you care about: does the model use the retrieved context, or does it ignore it and hallucinate? For agents, track: how many steps until it finishes? How many tool calls fail? How often does it get stuck in a loop? These aren’t benchmark metrics. These are production metrics.

Use span-level tracing (OpenTelemetry, Langsmith, or Prompt Caching dashboards) to see which requests are slow, token-heavy, or wrong. Don’t wait for user complaints.

The truth about eval frameworks

Braintrust, Promptfoo, DSPy, Ragas—they’re all solving real problems, but none are necessary to start. You need exactly one test file that runs your baseline cases and compares latency and cost against your last deploy. That’s the floor. Use these frameworks later when you have 100+ cases and need systematic tracking, A/B testing across variants, or running evals in CI. But you don’t start there.

Bottom line: Build a 20–50 case regression test from your real data, measure accuracy and cost against it before every deploy, and add 5–10 adversarial cases that target your specific failure modes. This catches 80% of production issues without any framework at all.

Question via Hacker News