The Benchmark — BIG-Bench Hard
A plain-English explainer of one AI evaluation benchmark: what it measures, how it works, and when to trust it.
BIG-Bench Hard
A curated subset of the hardest tasks from Google’s BIG-Bench that require multi-step reasoning and remain challenging for the largest language models.
What it measures
BIG-Bench Hard (BBH) tests whether models can perform complex reasoning across diverse domains—logic puzzles, formal semantics, algorithmic tasks, and implicit commonsense inference. The benchmark is deliberately designed around tasks where scaling model size alone doesn’t guarantee success, prioritizing tasks where even 100B+ parameter models score below 80% on their first attempt.
Why it was created
Google released BIG-Bench in 2022 as an attempt to move beyond saturated benchmarks like MMLU, which were showing ceiling effects. They identified that many benchmarks had stopped discriminating between capable models. BBH specifically extracted 23 hard tasks from the full BIG-Bench suite in late 2022 to create a tighter, more challenging benchmark that would remain useful as models improved.
How it works
BBH contains exactly 23 tasks with 10 examples each (230 problems total). Tasks span symbolic manipulation (formal logic, word sorting), knowledge reasoning (conceptual combinations, movie recommendations based on plot summaries), and algorithm simulation (computing basic operations on novel data structures). Scoring is exact-match accuracy—no partial credit. Most tasks use few-shot prompting (3-5 examples provided). Tasks include things like: “Is {X} a {Y}?” for compositional reasoning, or asking models to trace through simple algorithms manually.
What scores mean in practice
- Human expert baseline: ~89% (humans with task-specific domain knowledge)
- GPT-4 (original): ~83% (August 2023)
- Claude 3 Opus: ~86.8% (March 2024)
- Llama 2 70B: ~81.3%
- GPT-3.5: ~71%
- SOTA 2 years ago (late 2022): ~70-75% was considered strong
A score above 85% places a model in the “capable reasoning” tier. Below 75% suggests struggles with multi-step logic. The spread matters more than absolute numbers here—if a model jumps 15 points on BBH between versions, that’s a meaningful capability gain. The 23-task structure means each task represents ~4.3% of the score.
Known limitations
- Small sample size: Only 230 total examples means high variance. A model getting 2-3 additional questions right shifts the overall score by ~1%. This creates statistical noise and makes fine-grained ranking unreliable.
- Prompt sensitivity: BBH scores are extremely sensitive to how few-shot examples are selected and formatted. Chain-of-thought prompting can boost scores by 5-10 points, making “standard” reporting inconsistent across papers. You often can’t tell if an improvement is real or just better prompting.
- Ceiling risk emerging: By late 2024, frontier models (GPT-4, Claude 3.5) are approaching 90%+, reducing BBH’s discriminative power. It’s beginning to show the same saturation problem it was created to solve.
When to trust it (and when not to)
- Trust it when: Comparing two models on reasoning capability where both are in the 70-85% range. BBH remains useful for detecting real deficits in multi-step logic. It’s also good for checking whether a model has genuine reasoning ability vs. memorization (the diverse task types make that harder to game than single-domain benchmarks).
- Don’t trust it for: Ranking models above 85% (ceiling effects), claiming one model is definitively “better” without error bars (sample size is too small), or using a single run—results need multiple seeds. Also don’t use it in isolation; pair it with GPQA or MATH for a fuller picture of reasoning.