Where randomness meets reason

LLM Evaluation and Benchmarks: What They Measure, What They Miss, and How to Evaluate for Your Use Case How LLM evaluation benchmarks actually work, what they measure, what they miss, and how to build evaluation that matters for your specific application. 2026-07-06T12:00:00.000Z Deep Dives Deep Dives deep-divereferencearchitecture

LLM Evaluation and Benchmarks: What They Measure, What They Miss, and How to Evaluate for Your Use Case

How LLM evaluation benchmarks actually work, what they measure, what they miss, and how to build evaluation that matters for your specific application.

deep-divereferencearchitecture
The post you bookmark. One topic, covered end to end.

How LLM evaluation benchmarks actually work, what they measure, what they miss, and how to build evaluation that matters for your specific application.

LLM Evaluation and Benchmarks: What They Measure, What They Miss, and How to Evaluate for Your Use Case

Every model release comes with a table of benchmark scores. Anthropic publishes GPQA numbers for Claude Opus 4.8. Google highlights agentic coding benchmarks for Gemini 3.5 Flash. xAI claims Grok 4.3 leads on non-hallucination rates, though the specific evaluation methodology is not publicly detailed. These numbers shape purchasing decisions, architecture choices, and career reputations — but most practitioners can’t explain what MMLU actually tests, why HumanEval is a poor proxy for real coding ability, or how Chatbot Arena ratings are computed. That gap between “I saw the leaderboard” and “I understand the measurement” is where bad decisions happen.

This post covers the mechanics of major benchmarks, the statistical machinery behind arena-style ratings, the known failure modes of current evaluation, and a practical framework for building evaluations that reflect actual application requirements.

Table of Contents

The Benchmark Zoo: What Each One Actually Measures {#the-benchmark-zoo}

A benchmark is a dataset of questions with known answers, plus a scoring protocol. The dataset determines what domain is tested. The scoring protocol determines what counts as correct. Both matter enormously, and both introduce biases that are rarely discussed in model announcement blog posts.

Diagram

Major benchmark categories and what each family targets.

MMLU and Its Successors {#mmlu-and-its-successors}

MMLU (Massive Multitask Language Understanding) was introduced by Hendrycks et al. in 2021. It contains 15,908 multiple-choice questions across 57 subjects — from abstract algebra to world religions. Each question has four answer choices. Scoring is accuracy: percentage of questions answered correctly.

The format matters. Every MMLU question is multiple-choice with exactly four options. This means a random baseline scores 25%. It also means the benchmark cannot distinguish between “knows the answer” and “can eliminate wrong answers” — a model that understands nothing about virology but recognizes that three answer choices contain implausible phrasing can score well.

MMLU uses few-shot prompting by default (typically 5-shot): the model sees five example question-answer pairs before the test question. This reduces variance from prompt sensitivity but introduces its own bias — models that are better at in-context learning get an artificial boost on domains they haven’t actually memorized.

The 57 subjects are weighted equally in the aggregate score, regardless of question count per subject. This means a subject with 100 questions counts the same as one with 500. Subcategory breakdowns (STEM, Humanities, Social Sciences, Other) average their constituent subjects.

MMLU-Pro (introduced mid-2024) addressed several MMLU weaknesses. It expanded answer choices from 4 to 10, reducing the random baseline to 10%. It filtered out “trivially easy” questions and added more reasoning-heavy items. The result: model scores on MMLU-Pro are typically 15–30 percentage points lower than on MMLU for the same model. This spread makes it more useful for distinguishing between frontier models that have saturated MMLU (scoring 85%+).

Diagram

Evolution of MMLU benchmarks — each iteration addresses flaws in the previous version.

MMLU-Redux took a different approach. Rather than creating new questions, it re-annotated a subset of MMLU questions and found that roughly 6–10% of the original MMLU labels were incorrect. An error in the gold-standard answer is invisible in aggregate scores — it penalizes correct models and rewards incorrect ones. MMLU-Redux provides corrected labels for a subset, making it more reliable for that slice but less comprehensive overall.

HumanEval, MBPP, and SWE-bench {#humaneval-mbpp-and-swe-bench}

HumanEval (OpenAI, 2021) contains 164 Python programming problems. Each problem includes a function signature, a docstring describing the desired behavior, and a set of unit tests. The model generates a function body; if all unit tests pass, the problem counts as solved.

The standard metric is pass@k: the probability that at least one of k generated samples passes all tests. pass@1 is the most common report — it measures whether a single attempt succeeds. pass@10 and pass@100 are also reported but are less practically relevant (most applications don’t generate 100 completions per query).

HumanEval’s 164 problems are roughly LeetCode easy-to-medium difficulty. Many frontier models score above 90% pass@1, which limits HumanEval’s ability to differentiate between top-tier models. The problems are also exclusively Python, self-contained (no external dependencies, no multi-file projects), and algorithmic rather than systems-oriented.

MBPP (Mostly Basic Python Programming) is similar in spirit but contains ~1,000 simpler problems. It serves as a lower bar — useful for evaluating smaller or older models but not discriminating at the frontier.

SWE-bench (Princeton, 2023) is the benchmark that changed how the industry evaluates coding ability. It contains 2,294 real GitHub issues from 12 Python repositories (Django, Flask, Scikit-learn, etc.). Each task requires the model to read an issue description, understand a large codebase, and produce a patch that resolves the issue. The patch is evaluated by running the repository’s test suite.

SWE-bench Verified is a curated subset of ~500 problems where human annotators confirmed that the issue is solvable given the information available and the tests are deterministic. This is the more reliable variant.

The gap between HumanEval and SWE-bench scores reveals something important. A model scoring 95% on HumanEval might score 40–50% on SWE-bench Verified because SWE-bench requires codebase navigation, multi-file reasoning, and understanding existing test infrastructure — none of which HumanEval tests. Devstral 2 reaching 72.2% on SWE-bench Verified represents significant progress on this challenging benchmark of what was a 50+ point gap just a year ago.

Diagram

Code benchmarks in order of complexity — the jump from HumanEval to SWE-bench is where most models stumble.

GPQA: Graduate-Level Reasoning {#gpqa}

GPQA (Graduate-Level Google-Proof Q&A) targets questions that require genuine expert reasoning. The construction process was rigorous: domain experts (PhD students and researchers) wrote multiple-choice questions in biology, chemistry, and physics. Each question was validated by other experts in the same domain and by non-experts with internet access. A question qualified for the benchmark only if experts could answer it correctly but non-experts with Google could not.

The “Diamond” subset (198 questions) applies the strictest filtering criteria and is the most commonly cited variant. The broader GPQA set contains 448 questions.

GPQA Diamond scores remain the most discriminating benchmark for frontier reasoning models. Expert human performance on GPQA Diamond is approximately 65–75% (domain experts outside their specialization), while domain-matched experts score roughly 81%. Some frontier models are approaching or exceeding this range, which has prompted discussion about whether GPQA is approaching saturation for the strongest reasoning models.

MATH and GSM8K {#math-and-gsm8k}

GSM8K (Grade School Math 8K) contains 8,500 grade-school-level math word problems. Each problem requires 2–8 steps of arithmetic reasoning. Answers are numeric. This benchmark is largely saturated — frontier models score above 95%, and even mid-tier models clear 90%.

MATH (Hendrycks et al., 2021) is harder: 12,500 competition-level math problems across seven subjects (Prealgebra, Algebra, Number Theory, Counting & Probability, Geometry, Intermediate Algebra, Precalculus). Problems are drawn from AMC, AIME, and similar competitions. Difficulty levels range from 1 to 5.

The scoring protocol for MATH is exact match on the final answer, formatted in LaTeX. This is stricter than it sounds — a model that arrives at the correct numerical value but formats it as \frac{1}{2} instead of 0.5 (or vice versa) may be marked wrong depending on the evaluation harness. Different evaluation frameworks handle this differently, which is one reason MATH scores vary across papers for the same model.

AIME 2024/2025 problems have become an informal benchmark — these are recent competition problems that are less likely to be in training data. Model providers increasingly cite AIME performance, though there is no standardized evaluation protocol for it.

Truthfulness and Hallucination Benchmarks {#truthfulness-and-hallucination-benchmarks}

TruthfulQA (Lin et al., 2022) contains 817 questions designed to elicit falsehoods that humans commonly believe. Examples: “What happens if you crack your knuckles too much?” (common misconception: arthritis). The benchmark tests whether models reproduce popular misconceptions versus giving technically accurate answers.

The evaluation metric is the percentage of responses judged “truthful” by a fine-tuned frontier-model judge (or human annotators). This introduces meta-evaluation concerns — the quality of evaluation depends on the judge model.

HaluBench and related hallucination benchmarks measure whether models confabulate facts. These typically present a model with a context passage, ask it to answer a question, and check whether the answer is supported by the passage (closed-book hallucination) or factually correct (open-book hallucination).

xAI’s claim that Grok 4.3 leads on non-hallucination rate likely refers to evaluation on one of these benchmarks or an internal variant, though the specific methodology isn’t always disclosed in detail.

How Arena Ratings Work {#how-arena-ratings-work}

Chatbot Arena (LMSYS) is the most influential human-preference benchmark. The methodology is straightforward in concept: users submit a prompt, receive responses from two anonymous models, and pick the better response (or declare a tie). The results are aggregated into Elo-style ratings.

The actual rating system uses the Bradley-Terry model, not classical Elo. In the Bradley-Terry framework, each model has a latent “strength” parameter. The probability that model A beats model B is:

P(A > B) = exp(β_A) / (exp(β_A) + exp(β_B))

Maximum likelihood estimation fits the β parameters across all collected pairwise comparisons. The resulting scores are then linearly transformed to an Elo-like scale where the median model sits around 1000-1100.

Diagram

How Chatbot Arena produces ratings from pairwise human preferences.

Several important details affect interpretation:

User population bias. Arena users are not representative of all LLM users. They skew toward English-speaking, technically oriented individuals who enjoy testing models. Prompts tend toward creative writing, knowledge questions, and coding — not enterprise summarization or customer service.

Prompt distribution. The prompts users submit are not uniformly distributed across difficulty or domain. Easy prompts produce many ties, which dilute the signal. Very hard prompts may produce arbitrary preferences (the user can’t tell which answer is correct).

Category breakdowns. LMSYS provides arena ratings broken down by category (coding, math, creative writing, instruction following, etc.). These sub-ratings often tell a different story than the overall number. A model might rank #3 overall but #8 in coding and #1 in creative writing.

Confidence intervals. Arena ratings have confidence intervals, and they’re often wider than people assume. A 20-point difference between two models frequently has overlapping 95% confidence intervals, meaning the true ranking is uncertain.

MT-Bench (Multi-Turn Benchmark) is a related but distinct evaluation: 80 multi-turn conversations across 8 categories, judged by GPT-4-class models on a 1–10 scale. It’s faster and cheaper than arena-style evaluation but relies on LLM-as-judge, which introduces its own biases (see below).

What Benchmarks Get Wrong {#what-benchmarks-get-wrong}

Contamination and Overfitting {#contamination-and-overfitting}

Benchmark contamination is the most discussed failure mode. If MMLU questions appear in a model’s training data, the model can memorize answers rather than reason about them. This is difficult to detect conclusively — a model might have seen a paraphrased version of a question, or content from the same textbook the question was derived from.

Evidence of contamination is mostly circumstantial:

  • Models sometimes score anomalously well on specific MMLU subcategories while performing normally on adjacent ones
  • Performance on “canary” questions (newly created, never published) sometimes diverges sharply from performance on established benchmarks
  • Some models show unusually low variance across runs on benchmarks where questions are sampled

Overfitting without contamination is a subtler problem. A lab can optimize prompting strategies, chain-of-thought formatting, and decoding parameters specifically for benchmark evaluation without ever training on the test set. This is technically legitimate but makes benchmark scores less predictive of general capability.

The practice of reporting “best-of” results — running a benchmark multiple times with different prompts or sampling parameters and reporting the highest score — inflates numbers in ways that are invisible to readers comparing leaderboard entries.

Diagram

Three layers where benchmark scores can become misleading.

The Gap Between Benchmark and Production {#the-gap-between-benchmark-and-production}

Benchmarks test isolated capabilities. Production applications combine capabilities under constraints.

A RAG application needs a model to: read retrieved context accurately, avoid hallucinating beyond the context, synthesize information from multiple passages, follow formatting instructions, and do this within a latency budget at acceptable cost. No single benchmark measures this combination. A model that scores well on TruthfulQA, MMLU, and instruction-following benchmarks independently might still fail when these capabilities must operate simultaneously.

Instruction-following fidelity is one of the largest gaps. Benchmarks rarely test whether a model follows complex, nested formatting instructions — “respond in JSON with exactly three keys, where the ‘reasoning’ field contains at most 50 words.” IFEval (Instruction-Following Eval) exists for this purpose, but it covers a narrow slice of the instruction-following space.

Long-context utilization is another. MMLU tests are short — a few sentences of question text. Production applications frequently need models to reason over 10K–100K tokens of context. The “needle in a haystack” test measures recall at various context positions, but recall is a necessary rather than sufficient condition for useful long-context behavior. A model might find the needle but fail to synthesize it with other information in the haystack.

Robustness to input variation is rarely tested. Benchmarks present clean, well-formatted inputs. Production inputs contain typos, ambiguous phrasing, mixed languages, and adversarial patterns. A model that scores 90% on clean benchmark inputs might score 70% on the same questions with realistic noise added.

Benchmark CapabilityProduction RequirementGap
Multiple-choice accuracy (MMLU)Open-ended correct answersFormat mismatch
Single-function code gen (HumanEval)Multi-file codebase changes (SWE-bench)Scope mismatch
Short-context QA50K+ token document reasoningContext length
Clean, unambiguous promptsNoisy, underspecified user inputsRobustness
English-onlyMultilingual, code-switchingLanguage coverage
Single-turnMulti-turn with stateConversation management

Building Your Own Evaluation {#building-your-own-evaluation}

Generic benchmarks answer “how capable is this model in general?” Application-specific evals answer “how well does this model perform on my task?” The second question is the one that matters for engineering decisions.

Step 1: Define What “Good” Means

Start with the actual failure modes that would cause user-facing problems. For a medical summarization system, a hallucinated drug interaction is catastrophic; a slightly awkward sentence is acceptable. For a creative writing tool, the priority ordering reverses.

Write down 5–10 specific quality criteria, ordered by importance. Examples for a customer support bot:

  1. Factual accuracy relative to knowledge base (critical)
  2. Correct identification of customer intent (critical)
  3. Appropriate escalation when confidence is low (important)
  4. Tone consistency with brand guidelines (important)
  5. Response conciseness (nice to have)

Step 2: Build a Golden Dataset

A golden dataset contains input-output pairs where the expected output has been verified by humans. Size matters less than diversity and difficulty distribution.

Minimum viable eval set: 100–200 examples. This is enough to detect differences of ~10 percentage points between models with reasonable statistical confidence (see statistical rigor section below). For finer-grained comparisons, 500+ examples are needed.

Difficulty distribution. If 90% of examples are easy and 10% are hard, every model will score 85%+ and the eval won’t discriminate. Intentionally include edge cases, ambiguous inputs, and adversarial examples. A rough target: 30% easy, 40% medium, 30% hard.

Versioning. Golden datasets must be version-controlled. When a model update changes behavior, the eval dataset provides a fixed reference point. Store examples in a structured format (JSONL is standard) with metadata:

{"id": "cs-047", "input": "I was charged twice for order #4521", "expected_intent": "billing_dispute", "expected_action": "escalate_to_billing", "difficulty": "easy", "tags": ["billing", "duplicate_charge"]}
{"id": "cs-048", "input": "your product broke my other equipment do I get compensated", "expected_intent": "damage_claim", "expected_action": "escalate_to_claims", "difficulty": "hard", "tags": ["liability", "ambiguous_phrasing"]}

Step 3: Choose Evaluation Methods

Three approaches, in order of reliability:

Exact match / rule-based. If the expected output is a classification label, a number, or structured data with a defined schema, deterministic comparison works. Fast, cheap, reproducible. Use this wherever possible.

def eval_intent_classification(model_output: str, expected: str) -> bool:
    # Normalize and compare
    return model_output.strip().lower() == expected.strip().lower()

Code-based / functional tests. For code generation, run the output and check against test cases. For data extraction, validate against a schema. For SQL generation, execute the query and compare results.

def eval_sql_generation(generated_sql: str, expected_results: list[dict], db_conn) -> bool:
    try:
        actual = db_conn.execute(generated_sql).fetchall()
        return normalize_results(actual) == normalize_results(expected_results)
    except Exception:
        return False

LLM-as-judge. For open-ended text where multiple valid outputs exist — summarization, creative writing, conversational responses — use a strong model to evaluate a weaker model’s output. This is the most flexible but least reliable approach; see the dedicated section below.

Diagram

Basic evaluation pipeline — inputs flow through the model, outputs are scored against expectations.

Evaluation Architecture Patterns {#evaluation-architecture-patterns}

Pattern 1: Offline Batch Evaluation

Run the full eval suite against a model, collect all outputs, score them, produce a report. This is the simplest pattern and sufficient for most model selection decisions.

import json
from dataclasses import dataclass

@dataclass
class EvalResult:
    example_id: str
    input_text: str
    expected: str
    actual: str
    score: float
    latency_ms: float
    tokens_used: int

def run_eval_suite(
    model_client,
    eval_dataset: list[dict],
    scorer_fn,
    model_name: str
) -> list[EvalResult]:
    results = []
    for example in eval_dataset:
        start = time.time()
        response = model_client.generate(example["input"])
        latency = (time.time() - start) * 1000

        score = scorer_fn(response.text, example["expected"])
        results.append(EvalResult(
            example_id=example["id"],
            input_text=example["input"],
            expected=example["expected"],
            actual=response.text,
            score=score,
            latency_ms=latency,
            tokens_used=response.usage.total_tokens,
        ))
    return results

Pattern 2: Comparative A/B Evaluation

Run two models on the same eval set and compare directly. This controls for dataset difficulty — if one model scores 80% and another 75%, the 5-point gap is more interpretable than absolute scores because both faced identical inputs.

Diagram

A/B evaluation structure — both models receive identical inputs for controlled comparison.

Pattern 3: Continuous Monitoring Evaluation

Sample production traffic, run it through evaluation asynchronously, and track metrics over time. This catches drift — both model drift (when providers update models silently) and input drift (when user behavior changes).

# Pseudo-code for async production eval sampling
async def eval_sampler(request_stream, sample_rate=0.05):
    async for request, response in request_stream:
        if random.random() < sample_rate:
            # Run async evaluation
            eval_result = await evaluate_response(
                input=request.prompt,
                output=response.text,
                context=request.retrieved_context,
            )
            await metrics_store.record(eval_result)

Pattern 4: Multi-Dimensional Scoring

Score each output on multiple criteria independently rather than producing a single aggregate score. This reveals which model is better at which aspect of the task.

ModelAccuracyFormat ComplianceLatency p50Cost per 1K evals
Claude Opus 4.892%97%2,100ms$14.20
Claude Sonnet 587%95%890ms$3.40
GPT-5.590%93%1,400ms$8.60
Gemini 3.5 Flash85%91%340ms$1.80

Hypothetical multi-dimensional eval results. These numbers are illustrative, not from actual benchmarks.

This kind of table makes the tradeoff explicit. If format compliance is non-negotiable and latency matters, Sonnet 5 might be the right choice despite lower accuracy. If accuracy is paramount and cost is secondary, Opus 4.8 wins.

LLM-as-Judge: When Models Evaluate Models {#llm-as-judge}

Using a strong model to evaluate weaker models’ outputs is now standard practice. The technique works better than most practitioners expect — but has well-documented biases.

How to Structure Judge Prompts

A judge prompt needs: the original input, the model output being evaluated, the evaluation criteria, and a scoring rubric.

JUDGE_PROMPT = """You are evaluating the quality of an AI assistant's response.

## Input
{input}

## Response to Evaluate
{response}

## Evaluation Criteria
1. Factual accuracy: Does the response contain only verifiable facts?
2. Completeness: Does it address all parts of the question?
3. Conciseness: Is it free of unnecessary repetition or filler?

## Scoring
Rate each criterion from 1-5 where:
1 = completely fails
2 = mostly fails
3 = partially meets
4 = mostly meets
5 = fully meets

Respond in JSON:
{{"accuracy": <int>, "completeness": <int>, "conciseness": <int>, "reasoning": "<brief explanation>"}}"""

Known Biases in LLM Judges

Verbosity bias. LLM judges tend to prefer longer responses, even when the shorter response is more accurate and complete. Mitigation: include “conciseness” as an explicit criterion and instruct the judge that longer is not inherently better.

Position bias. When comparing two responses (A vs B), judges favor the response in position A (or B, depending on the model). Mitigation: run each comparison twice with swapped positions and only count agreements.

Self-preference bias. Models tend to prefer outputs from their own model family. If using Claude Opus 4.8 as a judge, it may systematically favor Claude-generated responses. Mitigation: use a judge from a different provider than the models being evaluated, or average across multiple judges.

Style over substance. Judges are influenced by formatting, markdown usage, bullet points, and confident tone — even when the content is wrong. Mitigation: instruct the judge to evaluate factual content separately from presentation.

Diagram

Position-debiasing through dual evaluation with swapped response ordering.

Multi-Judge Ensembles

Using multiple judge models and aggregating their scores reduces individual model biases. A practical ensemble:

JUDGES = [
    {"model": "claude-opus-4.8", "provider": "anthropic"},
    {"model": "gpt-5.5", "provider": "openai"},
    {"model": "gemini-3.5-flash", "provider": "google"},
]  # Example judges; substitute with your preferred models

async def ensemble_judge(input_text, response_text, criteria):
    scores = []
    for judge in JUDGES:
        score = await run_judge(
            judge_model=judge["model"],
            input_text=input_text,
            response_text=response_text,
            criteria=criteria,
        )
        scores.append(score)

    # Median is more robust than mean for small ensembles
    return {
        criterion: statistics.median([s[criterion] for s in scores])
        for criterion in criteria
    }

The cost of multi-judge evaluation adds up. For a 200-example eval set with three judges and position debiasing (2x runs per judge), that’s 1,200 judge calls. At typical frontier model pricing, this costs $5–$30 depending on response length and model choice. Gemini 3.5 Flash is a cost-effective judge option for high-volume evaluation given its speed advantage.

Statistical Rigor for Small Eval Sets {#statistical-rigor-for-small-eval-sets}

Most application-specific eval sets have 100–500 examples. At this scale, statistical significance matters and is often ignored.

Confidence Intervals for Accuracy

If a model scores 85% on 200 examples, the 95% confidence interval is approximately ±5 percentage points (using the Wilson interval):

from statsmodels.stats.proportion import proportion_confint

correct = 170
total = 200

lower, upper = proportion_confint(correct, total, alpha=0.05, method='wilson')
# lower ≈ 0.797, upper ≈ 0.893
# True accuracy is probably between 79.7% and 89.3%

This means a model scoring 85% and one scoring 82% on the same 200-example eval set are not distinguishable at this sample size. Claiming one is better than the other would be noise, not signal.

Paired Comparisons: McNemar’s Test

When comparing two models on the same eval set, use a paired test rather than comparing aggregate accuracies. McNemar’s test examines the 2x2 table of disagreements:

Model B CorrectModel B Wrong
Model A CorrectBoth right (a)A right, B wrong (b)
Model A WrongA wrong, B right (c)Both wrong (d)

The test statistic uses only the off-diagonal cells (b and c) — the cases where the models disagree. This is more powerful than comparing overall accuracy because it focuses on the informative cases.

from statsmodels.stats.contingency_tables import mcnemar

# Example: on 200 questions
# Both correct: 150, A only: 20, B only: 12, Both wrong: 18
table = [[150, 20], [12, 18]]
result = mcnemar(table, exact=True)
print(f"p-value: {result.pvalue:.4f}")
# If p < 0.05, the difference is statistically significant

Bootstrap Confidence Intervals for Complex Metrics

For metrics more complex than accuracy — F1 scores, BLEU, composite scores — bootstrap resampling provides confidence intervals without distributional assumptions:

import numpy as np

def bootstrap_ci(scores, n_bootstrap=10000, ci=0.95):
    bootstrapped_means = []
    for _ in range(n_bootstrap):
        sample = np.random.choice(scores, size=len(scores), replace=True)
        bootstrapped_means.append(np.mean(sample))

    lower = np.percentile(bootstrapped_means, (1 - ci) / 2 * 100)
    upper = np.percentile(bootstrapped_means, (1 + ci) / 2 * 100)
    return lower, upper

Minimum Sample Sizes

Detectable DifferenceRequired N (per model)Power (0.8)
20 percentage points~25Binary accuracy
10 percentage points~100Binary accuracy
5 percentage points~400Binary accuracy
3 percentage points~1,100Binary accuracy

These numbers assume comparing two independent proportions. Paired comparisons (same examples, different models) require fewer samples.

Evaluation in CI/CD {#evaluation-in-cicd}

Eval should run automatically when prompts change, retrieved context changes, or model versions update. The key design decisions:

What to Gate On

Hard gates (block deployment if eval fails):

  • Accuracy on critical categories drops below threshold
  • Any safety/toxicity test case fails
  • Structured output format compliance drops below 95%

Soft gates (alert but don’t block):

  • Overall accuracy drops by more than 2 percentage points
  • Average latency increases by more than 20%
  • Cost per query increases by more than 15%
Diagram

CI/CD evaluation flow — hard gates block deployment, soft gates generate alerts.

Practical Considerations

Cost control. Running 500 eval examples through a frontier model costs $5–$50 per run. If CI runs 20 times per day, that’s $100–$1,000/day. Solutions: use a cheaper model for initial screening (Gemini 3.5 Flash or Claude Haiku 4.5) and only run the full eval suite on merge to main.

Latency. A 500-example eval suite with 2-second average response time takes ~17 minutes serially. Parallelizing across 10 concurrent requests brings it to ~2 minutes. Most LLM APIs support this level of concurrency.

Determinism. Set temperature to 0 for eval runs. Even at temperature 0, some models are not perfectly deterministic (sampling can vary across different hardware configurations on the provider side), but variance is minimized.

Snapshot isolation. When comparing results across runs, ensure the eval dataset hasn’t changed. Pin the eval dataset version in CI configuration.

# Example CI config for LLM evaluation
eval:
  dataset: "evals/golden_v3.2.jsonl"
  models:
    - name: "claude-sonnet-5"
      provider: "anthropic"
      temperature: 0
  thresholds:
    accuracy_min: 0.85
    safety_pass_rate: 1.0
    format_compliance_min: 0.95
    max_latency_p95_ms: 3000
  concurrency: 10
  judge_model: "gemini-3.5-flash"

The Evaluation Taxonomy: Choosing What to Measure {#evaluation-taxonomy}

Different application types need different evaluation strategies. This matrix maps application categories to the most relevant evaluation approaches:

Application TypePrimary MetricsEval MethodBenchmark Proxy
Classification / routingAccuracy, F1, confusion matrixExact match—
Information extractionField-level precision/recallSchema validation—
Code generationpass@1, test pass rateFunctional testsHumanEval, SWE-bench
SummarizationFaithfulness, coverage, concisenessLLM-as-judge—
Open-ended QAAccuracy, completenessLLM-as-judge + spot checkMMLU, GPQA
ConversationalCoherence, helpfulness, safetyLLM-as-judge + Arena-styleMT-Bench
RAGAnswer correctness, attribution, no hallucinationDeterministic + LLM-judge—
Diagram

Choose evaluation method based on output structure — deterministic where possible, LLM-judge for open-ended, human review for critical decisions.

RAG-Specific Evaluation

RAG systems have a unique evaluation challenge: the answer quality depends on both retrieval quality and generation quality, and failures in either component can produce bad outputs.

Retrieval evaluation (independent of the LLM):

  • Recall@k: Does the correct document appear in the top k retrieved chunks?
  • MRR (Mean Reciprocal Rank): How high does the correct chunk rank?
  • Precision@k: What fraction of retrieved chunks are relevant?

Generation evaluation (given correct retrieval):

  • Faithfulness: Does the answer only contain information from the retrieved context?
  • Answer correctness: Is the answer factually right?
  • Answer relevance: Does the answer address the question?

End-to-end evaluation (retrieval + generation together):

  • The most practically useful but hardest to decompose when something fails.
Diagram

RAG evaluation happens at three levels — retrieval, generation, and end-to-end — each catching different failure modes.

Frameworks like RAGAS (Retrieval Augmented Generation Assessment) automate this decomposition using LLM-as-judge for each component. The tradeoff: it introduces judge model bias at every evaluation stage, compounding potential errors. Cross-validate RAGAS scores against human judgments on a subset of examples before relying on them.

Hallucination Detection Specifically

Hallucination in production is the failure mode that causes the most real-world damage. Evaluation approaches:

Attribution checking. For RAG systems, verify that every claim in the output can be traced to a specific passage in the retrieved context. This can be partially automated:

def check_attribution(response: str, context: str, judge_model) -> dict:
    """Check if each claim in response is supported by context."""
    prompt = f"""Given this context:
{context}

And this response:
{response}

For each factual claim in the response, determine if it is:
- SUPPORTED: directly stated or clearly implied by the context
- NOT_SUPPORTED: not found in or contradicted by the context

Respond in JSON: {{"claims": [{{"claim": "...", "verdict": "SUPPORTED|NOT_SUPPORTED", "evidence": "..."}}]}}"""

    result = judge_model.generate(prompt)
    return json.loads(result)

Consistency checking. Generate multiple responses to the same question and check for contradictions. If a model gives different answers to the same factual question across runs, at least some responses are hallucinated. This works at temperature > 0 only.

Known-answer probing. Include questions with verifiably correct answers in production traffic (synthetic canaries). Track whether the model gets these right over time. A decline signals model degradation or context contamination.

Putting It Together: An Evaluation Strategy Template {#evaluation-strategy}

For a team starting from zero, here is a phased approach:

Week 1: Define 5 quality criteria specific to the application. Write 50 eval examples covering easy, medium, and hard cases.

Week 2: Implement exact-match or functional evaluation for any structured outputs. Set up LLM-as-judge for open-ended outputs. Run the eval suite against 2–3 candidate models.

Week 3: Expand to 200 eval examples. Add statistical analysis (confidence intervals, paired tests). Integrate eval suite into CI to run on prompt changes.

Week 4: Set up production sampling (5% of traffic) with async evaluation. Create dashboards for accuracy, latency, cost, and hallucination rate over time.

Ongoing: Add new eval examples when production failures are discovered. Re-run model comparisons when new models are released. Review and update quality criteria quarterly.

The cost of this infrastructure is modest — primarily LLM API costs for judge evaluations ($50–$200/month for most applications) and engineering time for the initial setup. The cost of not having it — deploying a model that hallucinates 5% more often, or missing a regression when a provider silently updates their model — is typically much higher.

Summary {#summary}

MMLU measures multiple-choice knowledge recall across 57 subjects; it’s largely saturated at the frontier and its successor MMLU-Pro is more discriminating. HumanEval measures single-function Python generation; SWE-bench Verified measures real-world codebase modifications and is a far better proxy for production coding ability. GPQA Diamond tests expert-level reasoning and remains one of the few benchmarks where frontier models haven’t clearly surpassed expert human performance. Arena ratings use Bradley-Terry pairwise modeling and carry wider confidence intervals than leaderboard presentations suggest.

All public benchmarks suffer from contamination risk, protocol variance, and the gap between isolated capability measurement and integrated production performance. The most actionable evaluation is application-specific: golden datasets of 200+ examples, scored with deterministic methods where possible and LLM-as-judge ensembles where necessary, integrated into CI, and monitored in production.

LLM-as-judge works better than expected but has documented biases (verbosity, position, self-preference, style). Mitigate with position swapping, multi-judge ensembles, and explicit rubrics. For small eval sets, use paired statistical tests (McNemar’s) rather than comparing aggregate accuracy — two models scoring 85% and 82% on 200 examples are probably indistinguishable.

Build evaluation infrastructure early. It compounds in value: every production failure becomes a new eval example, every model migration becomes a quantified comparison rather than a vibes check.

Further Reading {#further-reading}