The Benchmark — LiveCodeBench

LiveCodeBench

A continuously updated benchmark that evaluates code generation models on real, practical programming problems collected from online coding platforms.

What it measures

LiveCodeBench tests a model’s ability to write working code that solves realistic programming problems—not toy exercises, but actual problems from LeetCode, AtCoder, and CodeForces. It measures both correctness (does the code pass test cases?) and practical utility (can a model solve problems it hasn’t seen in training?). The benchmark spans multiple programming languages and difficulty levels, from easy algorithmic problems to hard data structure and optimization challenges.

Why it was created

HumanEval, the standard code generation benchmark since 2021, became increasingly saturated and leaked into training data by 2023-2024. Researchers created LiveCodeBench (released in late 2023 by researchers including folks connected to the vLLM project) to address this: by pulling problems from live coding platforms and continuously adding new ones, it creates a moving target that’s much harder to overfit on or contaminate through training data scraping. It reflects actual coding interview difficulty rather than simplified academic problems.

How it works

LiveCodeBench pulls coding problems from three major sources: LeetCode, AtCoder, and CodeForces. Each problem includes a description, input/output constraints, and a set of hidden test cases. Models generate code in Python, Java, C++, or JavaScript. A solution is judged pass@1 (does the first attempt pass all test cases?) or pass@k (does at least one of k attempts pass?). The benchmark splits problems by difficulty tier and by date—“recent” problems are those added after the model’s training cutoff, providing a genuine generalization test. As of early 2024, the benchmark contains 300+ problems with regular monthly additions.

What scores mean in practice

As of Q4 2024, GPT-4 achieves ~85% pass@1 on easier problems and ~60% on hard ones. Claude 3.5 Sonnet scores similarly (~82% easy, ~58% hard). Open-source models like Llama 2 70B score around 50-60% on easy problems and 20-30% on hard ones. For context: human expert programmers (competitive coders) score near 95% on easy and 70-80% on hard. Two years ago (late 2022), GPT-3.5 scored roughly 48% on HumanEval; LiveCodeBench’s harder difficulty means current frontier models’ ~65% average represents real progress but also reveals that “solving code problems” is still far from solved.

Known limitations

• Contamination still possible: LiveCodeBench mitigates leakage, but problems are still sourced from public platforms. Training data cutoffs matter, and careful analysis of training corpora hasn’t fully ruled out indirect contamination through forums or tutorials discussing these exact problems.
• Test case brittleness: Correctness is binary (pass/fail on test cases), which doesn’t capture subtle bugs, inefficient solutions, or code that works but is unreadable. A solution passing all test cases doesn’t guarantee it would work on edge cases not in the suite.
• Language and domain skew: Heavily weighted toward algorithmic problems (LeetCode speciality) rather than systems programming, web development, or domain-specific code. Performance may not generalize to real-world codebases where models are actually deployed.

When to trust it (and when not to)

• Trust it for: Comparing code generation capabilities between models on a fresh, continuously-updated benchmark. If you’re choosing between two models specifically for algorithmic problem-solving, LiveCodeBench is far more reliable than the saturated HumanEval. It’s a genuine test of generalization.
• Don’t trust it alone for: Predicting whether a model can handle your production codebase. This benchmark tests isolated problem-solving; real engineering requires debugging, refactoring, understanding legacy systems, and writing idiomatic code in domain-specific contexts that LiveCodeBench doesn’t measure.