The Benchmark — FrontierMath

FrontierMath

A benchmark of 250 difficult, original mathematics problems designed to challenge frontier AI models on genuine unsolved-style problems rather than test-set memorization.

What it measures

FrontierMath evaluates whether AI models can solve genuinely difficult, previously unseen mathematics problems that require deep reasoning and domain expertise. The benchmark spans pure mathematics (number theory, combinatorics, algebra, geometry) and applied mathematics (physics, optimization), with problems ranging from Olympiad-level to early graduate coursework difficulty.

Why it was created

Existing math benchmarks like MATH (which features 12,500 problems from high school competitions) had become saturated—GPT-4 and other frontier models were approaching 95%+ accuracy by 2024. Epoch AI and collaborators created FrontierMath in 2024 to address this ceiling effect and provide a benchmark where contamination risks (models having seen the problems during training) are minimal, since the problems were newly created.

How it works

FrontierMath contains 250 original problems written by mathematicians and domain experts, with solutions verified by independent checkers. Problems are presented in natural language with mathematical notation. Scoring is binary per-problem: a solution is marked correct if the final numerical or symbolic answer matches exactly, with some allowance for equivalent forms (e.g., simplified fractions). The benchmark is deliberately small compared to MMLU (11,000+ items) or MATH (12,500 items)—this makes each problem statistically weighted and means a few failures significantly impact the overall score. No partial credit is awarded; intermediate work is not evaluated, only the final answer.

What scores mean in practice

As of early 2024, Claude 3.5 Sonnet achieved ~53% accuracy, and GPT-4o achieved ~50%. These scores represent a dramatic drop from the 90%+ accuracies these same models achieve on MATH. A human expert mathematician (PhD-level) would likely score 75–85% given unlimited time; the benchmark is calibrated to be genuinely difficult even for specialists. A score of 40% represents frontier-level performance; 25% is roughly where strong general-purpose models like Llama 2 sit. Two years ago (2022), the best models would have scored ~5–10% on unseen problems of this difficulty.

Known limitations

• Extreme smallness creates high variance: With only 250 problems, random fluctuations matter more. A model’s true capability might differ by 5+ percentage points from a single run; this makes score comparisons within a few percentage points unreliable.
• Verification bias and answer-format brittleness: Correct solutions in unexpected forms (e.g., an equivalent but non-canonical representation) may be marked wrong. The verification process, while careful, is still human-dependent and not exhaustively published.
• No insight into reasoning quality: The benchmark only checks final answers. A model could get lucky on a multi-step problem or fail on a simple arithmetic step in an otherwise sound proof. It doesn’t measure whether the model actually understands the underlying mathematics.

When to trust it (and when not to)

• Trust it when: You want to assess whether a model is advancing the frontier of mathematical reasoning beyond what existing benchmarks can measure. It’s most useful for distinguishing frontier models from strong baseline models in the 40–80% range, and for understanding contamination risk (new problems are hard to memorize).
• Don’t trust it when: Comparing models that differ by small margins (e.g., 51% vs 53%—this is noise, not a signal). Also be skeptical if you need fine-grained diagnostic feedback on what kinds of math a model struggles with; FrontierMath doesn’t provide that. It’s a ceiling test, not a diagnostic tool.