Office Hours — What specific metrics and evaluation methods work best for measuring LLM output quality in your domain?

What specific metrics and evaluation methods work best for measuring LLM output quality in your domain?

This question is harder than it looks because “quality” isn’t one thing. You need to measure different attributes depending on whether you’re building search, customer support, code generation, or compliance documentation. And the metrics that matter in development almost never match what matters in production.

The Three Evaluation Layers

Most teams conflate three separate problems. First, there’s benchmark performance (MMLU, SWE-Bench, Arena leaderboards)—useful for deciding which model to buy, but almost useless for knowing if your system actually works. Second, there’s offline evaluation on your own domain data—closer to reality, but still artificial because there’s no user on the other end. Third, there’s production telemetry—what actually matters, but it’s messy and delayed.

Start by being honest about which layer you’re actually measuring. Benchmark scores tell you almost nothing about whether Claude Opus 5 or GPT-5.6 Sol will be better at your specific task. The Remote Labor Index (Center for AI Safety) tracks real freelance jobs at professional quality, and the top agent still succeeds on only ~16% of tasks, which means whatever your benchmark says, assume your production success rate is lower.

Domain-Specific Metrics That Actually Track Reality

For customer support tickets, measure resolution rate (did the agent close it without escalation?) and customer satisfaction (did they rate it 4+ stars?). These are outcome metrics. Avoid measuring “response coherence” or “factuality” in isolation—a perfectly coherent hallucination still fails.

For code generation, measure test pass rate (does the generated code run and pass your test suite?) and security scan results (does it introduce vulns?). For coding agents specifically, track PR merge rate and whether the merged code survives production for 30 days. Databricks found that on its million-line codebase, GLM-5.2 matched Claude Opus 4.8 on test pass rate while costing significantly less—but they only discovered this by benchmarking on their own code, not on public leaderboards.

For RAG systems, measure end-to-end accuracy: does the LLM answer the user’s question correctly using retrieved documents? Don’t measure retrieval precision and LLM accuracy separately—that’s a useful debugging step, but the user cares about the final answer. If your retriever is perfect but the LLM hallucinates anyway, your system is broken.

For financial or legal document extraction, measure precision (of fields that are extracted, how many are correct?) and recall (of all correct fields, how many did you find?). But also measure the cost of human review—if your LLM is 95% accurate but requires a human to check every output, you may not have saved labor.

The Production Trap: Why Offline Evals Lie

The UK AI Security Institute (AISI) found that agent success on SWE-Bench Pro rises ~25% when you increase the test-time token budget from 1M to 10M. This means your offline benchmark result depends entirely on how much compute budget you’re willing to spend in production. A model that looks 85% capable at 1M tokens might be 94% capable at 10M tokens. You need to measure this explicitly on your workload.

Concurrent request handling also changes everything. A model that serves beautifully in isolation can hallucinate or contradict itself when handling 20 requests in parallel. Measure single-request performance and batch performance separately.

Custom Evals: The Real Work

Every team building production LLM systems needs hand-evaluated evals on their own data. Here’s a concrete starting point:

1. Sample 100–200 real examples from production (or your expected domain).
2. Generate outputs from your current model(s).
3. Have two humans score each output on: correctness (binary or 1–5 scale), usefulness (would a user find this helpful?), and safety (any red flags?).
4. Calculate inter-rater agreement (Cohen’s kappa). If it’s below 0.7, your rubric is ambiguous—refine it.
5. Track agreement rate and average score over time as you iterate.

This isn’t scalable beyond a few hundred examples, which is fine—it’s meant to catch major regressions and validate that model upgrades actually help. Automate the obvious stuff (test pass/fail for code, exact match for structured fields) but always keep humans in the loop for judgment calls.

What Breaks in Production That Evals Don’t Catch

The Daily Signal (August 3) highlighted the clearest signal: IBM found 92% of AI security breaches had basic access control failures—the vulnerability wasn’t in the LLM, it was in the infrastructure around it. Your eval might show the model generates good recommendations, but if you’re not logging outputs, not monitoring for toxic behavior, and not rate-limiting user requests, you have a problem.

Also measure edge cases explicitly. If your model handles typical queries fine but fails on anything longer than 2,000 characters, or whenever a field contains special characters, your evals missed it. Synthetic adversarial examples help: deliberately construct queries that are slightly weird—typos, unusual formatting, requests in a second language, edge cases your retriever might miss.

The Cost Metric You’re Probably Ignoring

For production systems, cost per successful outcome is more useful than cost per token. If GPT-5.6 Sol costs 3x more per token than Claude Sonnet 5 but solves your problem 2x faster and with fewer retries, the Sonnet might be cheaper end-to-end. Cursor’s agent architecture (planning with a frontier model, execution with cheaper models) successfully reduced costs by separating where you actually need capability from where you don’t.

Measure this explicitly: total cost (tokens + infrastructure) divided by successful outputs. This forces you to account for retry loops, escalations, and human review time—the parts that destroy your unit economics.

Avoiding the Audit Trap

Current LLM safety audits have massive blind spots (as the Daily Signal noted on August 4). Don’t outsource your entire evaluation to a vendor audit or a third-party framework. These are useful but insufficient. You need custom evaluation on the specific risks your system can create: hallucinations in your domain, harmful outputs for your use case, biased recommendations for your users.

Bottom line: Build three layers of evals—production telemetry on what actually matters (resolution rate, user satisfaction, merge rate), custom hand-evaluated evals on 100–200 representative examples from your domain, and explicit cost-per-outcome tracking. Benchmark scores tell you which model to start with; everything else is domain-specific and requires measuring on your own data.

Question via Hacker News