The Benchmark — MBPP (Mostly Basic Python Problems)
A plain-English explainer of one AI evaluation benchmark: what it measures, how it works, and when to trust it.
MBPP (Mostly Basic Python Problems)
Evaluates whether language models can write correct, executable Python code for practical programming tasks of low-to-medium difficulty.
What it measures
MBPP tests code generation across ~1,000 short Python problems ranging from data manipulation to basic algorithms. Unlike HumanEval (which focuses on algorithmic rigor), MBPP emphasizes functional correctness on “real-world-ish” tasks that practicing developers might encounter. Tasks include string formatting, list operations, number theory, and simple data structures—intentionally avoiding the hardest algorithmic puzzles.
Why it was created
HumanEval, released in mid-2021, became the default code-generation benchmark but covered only 164 problems, raising concerns about overfitting and limited scope. Google released MBPP in June 2021 to provide a larger, more diverse evaluation set that better represented the distribution of practical programming work. The intent was a complementary benchmark that would catch models that memorized HumanEval but struggled with slightly broader, less algorithmic code tasks.
How it works
MBPP contains 974 problems in the full set (427 in the “sanitized” version used most often). Each problem includes: a function signature, natural language description, 2–3 test cases, and a reference solution. Models generate Python code; evaluation runs the generated code against hidden test cases and checks for exact output match. Scoring is pass@1 (does it work on first try?) and pass@k (does it work within k samples?). A problem passes only if all test cases execute correctly—partial credit doesn’t exist.
What scores mean in practice
Human baseline: Experienced programmers achieve ~88% pass@1 on the sanitized MBPP set (the benchmark designers tested this).
Current leading models (late 2024):
- Claude 3.5 Sonnet: ~92% pass@1 on sanitized MBPP
- GPT-4: ~84–86% pass@1
- Llama 2 70B: ~50–55% pass@1
Historical context: In mid-2022, state-of-the-art was around 70% (Codex). Two years ago (late 2022), 75–80% would have been competitive.
The human baseline being ~88% means the benchmark is almost saturated with frontier models, but not quite. This is different from HumanEval, where top models now exceed 95%.
Known limitations
- Near-saturation: With leading models at 92%, MBPP is losing discriminative power. It will soon struggle to differentiate between strong models, making it less useful for tracking progress.
- Test case brittleness: Problems often have only 2–3 test cases, allowing models to pass with code that works on those specific inputs but fails on edge cases. A model that generates
return x + yfor “sum two lists” might pass if test cases don’t include empty lists. - Shallow capability measurement: The “mostly basic” framing means MBPP doesn’t stress-test complex reasoning, system design, or algorithms at scale. A model can be weak at real software engineering and still score well here.
When to trust it (and when not to)
- Trust it for: Comparing models in the 50–85% range, or checking whether a small model has basic code-generation competence. It’s still useful as a sanity check for whether a model can write runnable Python at all.
- Don’t trust it for: Distinguishing between frontier models (Claude vs. GPT-4 vs. newer competitors). At >90%, differences are noise, and you need harder benchmarks like LiveCodeBench, SWE-bench, or real code-review data. Also don’t use it to claim a model is “production-ready” for software engineering—passing MBPP is necessary but nowhere near sufficient.