The Benchmark — WinoGrande

WinoGrande

A pronoun resolution benchmark with 44,000 multiple-choice questions designed to test whether models can disambiguate which noun a pronoun refers to using commonsense reasoning.

What it measures

WinoGrande tests commonsense understanding by requiring models to resolve ambiguous pronouns in English sentences. Each question presents a sentence with two possible noun referents and asks which one a pronoun (“it,” “she,” “they”) refers to—but the correct answer requires real-world knowledge, not just syntax. For example: “The city councilmen refused the demonstrators a permit because they feared violence” — does “they” refer to the councilmen or demonstrators? Syntax alone won’t tell you; you need to know that city officials are typically the ones making permit decisions.

Why it was created

WinoGrande was released in 2020 by Salesforce Research to address a longstanding limitation in NLP evaluation: most benchmarks relied on surface-level pattern matching rather than genuine reasoning. The Winograd Schema Challenge (which inspired it) had been around since 2011 but was small—only 273 examples. Researchers wanted a large-scale version that could actually differentiate between models that memorize correlations and those that reason about meaning.

How it works

The benchmark contains 44,038 multiple-choice questions split into XL (5,600 questions, used for standard reporting) and Debiased (9,248 questions, with adversarial bias removal). Each question shows a sentence with one ambiguous pronoun and two possible antecedents. The model must choose which noun the pronoun refers to. Scoring is simple: percent correct. The dataset covers diverse domains—fiction, news, social media—to reduce dataset-specific artifacts. Questions were constructed semi-automatically from templates and then human-validated.

What scores mean in practice

Human accuracy on WinoGrande XL is approximately 94-96%, establishing a reasonable ceiling. GPT-4 scores around 87-89% on the XL set (as of late 2023). Claude 3 Opus achieves ~88%. Two years ago (2022), SOTA was around 82-84% (T5-11B fine-tuned). This means current top models are still 5-8 points below human performance—a meaningful gap. On the Debiased version (specifically designed to remove spurious correlations), scores are typically 2-3 points lower, suggesting models still exploit dataset biases even on this “harder” variant.

Known limitations

• Artificial task structure: Real pronoun resolution in natural text is different from multiple-choice disambiguation. Models may succeed by pattern-matching question format rather than true linguistic understanding, especially since one distractor noun is almost always wrong by convention.

• Limited reasoning complexity: While these questions need commonsense knowledge, they don’t require multi-step reasoning or integration of complex world models. A model that has seen similar examples in training can succeed through shallow analogy.

• Debiasing may be incomplete: The Debiased version attempts to remove statistical shortcuts, but human-annotated alternatives can introduce their own biases. Models that perform well on both versions may just have learned different spurious correlations rather than robust reasoning.

When to trust it (and when not to)

• Trust it for: Comparing two models’ ability to handle basic pronoun resolution at scale. A substantial gap (5+ points) suggests real differences in commonsense grounding or training data quality. Useful as a baseline signal in a battery of tests.

• Don’t rely on it alone for: Claims about “commonsense reasoning” or “understanding.” A high WinoGrande score doesn’t mean a model reasons well about causality, intentions, or social dynamics. Pair it with GPQA, BIG-Bench Hard, or more complex reasoning benchmarks to build confidence.