The Benchmark — ARC-Challenge (AI2 Reasoning Challenge)

ARC-Challenge (AI2 Reasoning Challenge)

A multiple-choice science benchmark requiring multi-step reasoning, designed to test reading comprehension and knowledge integration rather than simple fact retrieval.

What it measures

ARC-Challenge tests whether models can answer grade-school science questions (grades 3-9) that require genuine reasoning, not just pattern matching. Unlike benchmarks that reward memorization, these questions intentionally exclude straightforward answers in the text and require combining concepts across sentences or paragraphs. The dataset includes questions from actual standardized tests, filtered for difficulty.

Why it was created

AI2 created ARC in 2018 to identify a benchmark that couldn’t be solved by simple retrieval or surface-level pattern matching. At the time, models were showing inflated performance on reading comprehension tasks by exploiting shallow correlations between questions and documents. AI2 wanted a benchmark that genuinely tested reasoning — the kind humans need to pass science exams.

How it works

ARC-Challenge contains 1,119 multiple-choice questions, each with four answer options. The benchmark is filtered from a larger pool to include only questions that are difficult for retrieval-based baselines and simple language models — typically questions where at least 10% of humans get it wrong or where the right answer doesn’t appear as keywords in source documents. Scoring is straightforward: accuracy as a percentage of correctly answered questions. There’s also an “ARC-Easy” variant with 5,197 easier questions, but ARC-Challenge is the standard.

What scores mean in practice

Human expert performance on ARC-Challenge is roughly 94-97%. Claude 3.5 Sonnet currently scores around 96%, GPT-4 scores 96%, and open-source models like Llama 3.1 70B score around 85-87%. Two years ago (late 2022), SOTA was around 85% — GPT-3.5 and similar models achieved this range. The benchmark has become somewhat saturated at the top, with leading models essentially at human parity.

The gap between 85% and 96% is real but narrow: those remaining 11 percentage points often involve questions where even humans reasonably disagree or where truly obscure science knowledge matters.

Known limitations

• Saturation and contamination concerns: ARC-Challenge has been heavily cited and incorporated into training datasets. There’s documented evidence that recent models may have memorized portions of it, making benchmark scores less meaningful than they appear. By 2024, the benchmark no longer clearly separates strong models.

• Grade-school science is narrow scope: The benchmark tests a specific domain (science facts + basic reasoning) and doesn’t cover mathematics, coding, instruction-following, or many other capabilities. A model can score 96% on ARC-Challenge and still struggle with novel reasoning in other domains.

• Multiple-choice ceiling hides failure modes: Four-option multiple choice makes random guessing worth 25%. A model can get lucky, and the benchmark doesn’t capture whether reasoning is sound or just numerically correct. Open-ended evaluation would be more informative.

When to trust it (and when not to)

• Trust it for: Comparing models released before 2023, or comparing open-source and smaller models where scores are still spread out (75-90%). It’s still useful for identifying models that have fundamental reading comprehension issues.

• Don’t rely on it for: Distinguishing between state-of-the-art closed-source models in 2024+, or predicting performance on novel reasoning tasks. If you’re evaluating whether your model is “smart enough,” ARC-Challenge won’t tell you much if most competitors score 95%+.