The Benchmark — HumanEval

HumanEval

Measures whether language models can write functional Python code to solve short programming problems.

What it measures

HumanEval tests code generation capability by asking models to implement functions from docstrings and passing test cases. It focuses on basic algorithmic and string manipulation tasks—things like “write a function that finds the largest number in a list” or “implement a function to check if a number is a palindrome.” The benchmark doesn’t test system design, optimization, or real-world software engineering; it’s narrowly scoped to whether a model can produce syntactically correct, logically sound code for well-specified small problems.

Why it was created

When OpenAI released this benchmark in August 2021, code generation from language models was largely untested and unquantified. Researchers needed a simple, reproducible way to measure whether models like Codex could write working code. The 164-problem dataset was hand-written by researchers and designed to have clear pass/fail semantics via unit tests, making it objective and reproducible in a way that subjective code-quality judgments weren’t.

How it works

The benchmark contains 164 Python functions across multiple difficulty levels. Each problem provides a function signature, a docstring, and 1–10 hidden test cases. A model generates code; the code is executed against the test cases, and it either passes all of them (success) or fails (no partial credit). The metric is simple: percentage of problems solved. Standard methodology samples multiple completions per problem (typically 1 or 100) to account for randomness; the pass@k metric reports the probability that at least one of k samples passes.

What scores mean in practice

• Human performance: Expert programmers achieve ~99% on HumanEval.
• Current leading models: Claude 3.5 Sonnet and GPT-4o score around 92–95% (pass@1).
• 2022 SOTA: Codex (the original code model) scored ~28% pass@1.
• 2023 inflection point: Models like GPT-3.5 and early Claude variants hit 70–80%, marking a transition from “barely functional” to “often works.”

Pass@100 scores are typically 20–30 percentage points higher than pass@1, reflecting that models generate correct solutions but need multiple tries. A 90% pass@1 model is practically useful for simple tasks; below 70% becomes frustrating.

Known limitations

• Dataset saturation: The 164 problems are well-known and likely present in training data for newer models. Scores may reflect memorization rather than generalization, especially for models trained after 2023.
• Narrow problem scope: All problems are toy algorithms (sorting, string manipulation, recursion). The benchmark doesn’t test API usage, file I/O, debugging, refactoring, or any real-world complexity. A model can score 95% and still struggle writing production code.
• No partial credit for approach: If a solution has the right algorithm but an off-by-one error, it fails completely. This incentivizes models to be conservative rather than creative, and doesn’t reflect how humans evaluate code in practice.

When to trust it (and when not to)

• Trust it for: Comparing relative code generation ability between model versions or families. It’s a lightweight, reproducible smoke test. If model A scores 80% and model B scores 60%, A probably generates better code on simple tasks.
• Don’t trust it as: Evidence that a model is “production-ready” or can handle real codebases. High HumanEval scores tell you almost nothing about ability to refactor legacy code, write tests, or integrate with existing systems. It’s useful as one signal among many, not as a standalone capability predictor.