The Benchmark — HellaSwag

HellaSwag

Evaluates whether a model can predict the correct ending to a video scenario from multiple plausible choices, testing commonsense reasoning about physical and social situations.

What it measures

HellaSwag tests commonsense understanding of how everyday activities play out in the real world. Rather than abstract reasoning or factual knowledge, it requires models to understand physical causality, social norms, and practical consequences. The benchmark includes scenarios like cooking, sports, DIY repairs, and social interactions—situations where humans intuitively know what happens next, but which demand genuine understanding rather than pattern matching.

Why it was created

Released by Rohrbach et al. in 2019, HellaSwag was designed to address a gap in existing benchmarks: multiple-choice QA datasets had become saturated with high scores from simple statistical patterns. The creators noticed that models were gaming benchmarks like SWAG by learning surface-level correlations rather than demonstrating real reasoning. They specifically engineered HellaSwag to be adversarially filtered—wrong answer choices are generated by a neural model, then filtered to remove implausible distractors, creating hard negatives that require actual understanding.

How it works

The benchmark contains 10,042 multiple-choice questions, each with a video scenario (a few seconds of footage from ActivityNet or YouCook2), a partial caption describing the beginning of an activity, and four possible completions. One completion is correct; the other three are plausible-sounding but wrong. Scoring is simple: accuracy percentage. The filtering process is the key innovation—distractors aren’t randomly chosen; they’re adversarially generated to be hard to distinguish from correct answers without understanding the scenario.

What scores mean in practice

Human performance on HellaSwag is approximately 95-98% accuracy (the benchmark is designed to be solvable by humans who watch the video). As of late 2024, frontier models like GPT-4o and Claude 3.5 Sonnet score around 96-98%, while older models like GPT-3.5 scored ~80%. Two years ago (late 2022), SOTA was around 88-90%.

What’s striking: despite near-human performance from top models, HellaSwag hasn’t become trivial the way SWAG did. The adversarial filtering appears to have worked—the benchmark has stayed challenging relative to general capability progress.

Known limitations

• Video understanding is minimal: Models don’t actually process the video frames; they see only the caption describing the scenario start. This means the benchmark tests language-based commonsense reasoning about described situations, not genuine visual understanding. A model that never “sees” video can still score highly.

• Saturation approaching for frontier models: While HellaSwag has aged better than earlier benchmarks, GPT-4o and similar models are now approaching ceiling performance (96-98%), reducing its discriminative power for comparing state-of-the-art systems. Further progress in reasoning will be hard to measure here.

• Cultural and activity bias: The scenarios are drawn from Western internet videos (ActivityNet, YouCook2). Commonsense about cooking, sports, or household tasks varies globally, and models optimized on English web text may have skewed understanding of what’s “obvious” in other contexts.

When to trust it (and when not to)

• Trust it for: Evaluating whether a model has absorbed reasonable commonsense about everyday physical and social situations. It’s a useful sanity check that a model isn’t completely disconnected from how the world works. It’s also still useful for comparing models in the 75-92% range, where differentiation is possible.

• Don’t trust it for: Assessing frontier model capabilities or ranking GPT-4o vs Claude 3.5—both max out here, so HellaSwag gives no signal. It also doesn’t measure visual reasoning despite the video framing; it’s really a language benchmark in disguise.