The Benchmark — AIME (American Invitational Mathematics Examination)
A plain-English explainer of one AI evaluation benchmark: what it measures, how it works, and when to trust it.
AIME (American Invitational Mathematics Examination)
A benchmark of 30 challenging pre-calculus competition math problems where models must produce exact integer answers between 0 and 999.
What it measures
AIME tests mathematical reasoning, problem-solving, and symbolic manipulation at a high school competition level. Problems require multi-step reasoning, often combining algebra, geometry, number theory, and combinatorics. Unlike standardized tests with multiple choice, AIME demands exact answers with no partial credit—you either solve it correctly or you don’t.
Why it was created
AIME was designed in 1983 by the Mathematical Association of America (MAA) as a competition for advanced high school students. In AI evaluation, it’s been repurposed since 2023-2024 to test whether language models can solve problems that require sustained logical reasoning and aren’t easily solvable by pattern matching alone.
How it works
Each AIME benchmark typically includes 30 problems from past competitions (1983–present). Problems are presented in text form; the model must generate a numerical answer between 0 and 999. Scoring is binary: correct answer = 1 point, wrong answer = 0 points. Total possible score is 30. The benchmark tests pure mathematical ability without the confound of multiple-choice recognition. A single problem might require 5–10 reasoning steps across different domains.
What scores mean in practice
- Human baseline: Top competition math students score 15–24/30. Perfect scores (30/30) are rare even among exceptional competitors.
- Leading models (Jan 2024): GPT-4 scores around 19–21/30. Claude 3.5 Sonnet reaches ~20/30. Open-source models (Llama, Mistral) typically score 2–8/30.
- 2022 baseline: No models were systematically evaluated on AIME then; this is a recent benchmark adoption.
- Scale context: Scoring 12/30 is roughly average for AIME participants; 20+/30 is elite-tier reasoning.
Known limitations
- Tiny dataset with no train/test separation: Only 30 problems per benchmark run. This is orders of magnitude smaller than MMLU (14K questions). Results are noisy; a model’s true capability could differ significantly from one 30-problem sample.
- Potential contamination in training data: AIME problems have been public since 1983 and widely discussed online. It’s unclear whether GPT-4 and Claude have seen these exact problems or similar ones during training, making reported scores hard to interpret.
- Doesn’t test all mathematical reasoning types: Focuses on competition math, which favors symbolic manipulation and clever tricks. It doesn’t evaluate applied math, statistics, proof writing, or explanation quality—only final answers.
When to trust it (and when not to)
- Trust it for: Comparing models on high-stakes mathematical reasoning. If you care whether a model can solve genuinely hard math problems, AIME is a real test. It’s harder to game than multiple-choice benchmarks.
- Don’t trust it for: Drawing strong conclusions from single runs (30 problems is statistically thin). Don’t use it to claim one model is definitively better than another without multiple independent evaluations. Be skeptical of scores >22/30 given contamination risk.