The Benchmark — DROP (Discrete Reasoning Over Paragraphs)

DROP (Discrete Reasoning Over Paragraphs)

Evaluates whether language models can extract and manipulate numerical information from text to answer complex reasoning questions.

What it measures

DROP tests a model’s ability to read a paragraph, locate relevant numbers, and perform discrete reasoning operations—addition, subtraction, sorting, counting—to derive an answer. Unlike MMLU or ARC, which test broad knowledge, DROP focuses on a narrow but crucial skill: can the model understand when how to use numerical facts rather than just recall them? Tasks include questions like “How many more rushing yards did player X have than player Y?” or “What was the average score across these games?”

Why it was created

Released by Allen AI in 2019, DROP was designed to address a blind spot in existing reading comprehension benchmarks. Models were scoring well on SQuAD and similar datasets by pattern-matching, but struggled with paragraphs containing numbers. The creators wanted to measure genuine reading and arithmetic reasoning—skills that seemed elementary to humans but were breaking points for models.

How it works

The dataset contains 96,567 questions over 55,000 Wikipedia and news passages. Each question requires extracting numbers from text and performing operations (arithmetic, sorting, counting) to reach the answer. Questions are open-ended with string-match evaluation—the system checks if the model’s answer numerically equals the correct answer (e.g., both output “42”). Scoring is F1 score over exact match and partial credit metrics. A single question might require identifying five numbers across a paragraph and summing three of them.

What scores mean in practice

Human performance: ~96% F1 (near-perfect, though humans make occasional arithmetic errors).

Current SOTA (early 2024): GPT-4 reaches ~92% F1; Claude 3 scores ~90%. Smaller open models like Llama 2 (70B) score ~80%.

2 years ago (early 2022): Strong performance was ~80% F1. The benchmark has saturated less than MMLU, but improvements have slowed as models approach human-level performance.

The meaningful distinction here is whether a model can reliably handle multi-step numerical reasoning. A score of 85%+ suggests competent arithmetic and attention to detail; below 75% indicates frequent errors in number extraction or operation selection.

Known limitations

Saturation approaching. With top models at 92%, there’s limited room to differentiate further. Benchmark ceiling is near human performance, leaving little signal for model comparison going forward.
Wikipedia/news bias. All passages come from structured domains with clear numerical context. Real-world numerical reasoning—financial documents, scientific papers with ambiguous units, tables with implicit assumptions—is underrepresented.
Single-answer focus. DROP assumes one correct numerical answer. Many real reasoning problems have multiple valid approaches or answers (e.g., “roughly 50 million” vs “47-53 million”). The strict matching penalizes reasonable variations.

When to trust it (and when not to)

Trust it for: Evaluating whether a model can reliably extract and manipulate numbers from text. If you’re building a system that needs to answer “How much did revenue grow?” from quarterly reports, DROP scores are directly relevant.
Don’t rely on it for: Predicting performance on complex reasoning involving uncertainty, estimation, or multi-document synthesis. A model that scores 88% on DROP might still fail on real financial analysis tasks where numbers must be weighted by confidence or adjusted for missing context.