The Benchmark — LongBench

LongBench

Evaluates how well language models perform on long-context reasoning tasks across diverse domains, measuring their ability to maintain coherence and extract information from documents up to 4,000+ tokens.

What it measures

LongBench tests a model’s ability to handle extended sequences of text—roughly 4,000 to 4,500 tokens on average, with some tasks exceeding 10,000 tokens. Rather than testing a single capability, it spans eight domains: single-document QA, multi-document QA, summarization, few-shot learning, code completion, synthetic tasks (like finding a specific number in a long list), and language understanding. The benchmark deliberately includes both natural long documents (papers, books, legal texts) and synthetic needle-in-haystack style tasks to separate real comprehension from pattern-matching.

Why it was created

As models began claiming support for longer context windows (4K, 8K, 128K tokens), there was no standardized way to evaluate whether they actually used those windows effectively or just claimed to support them. LongBench was released by LMSYS and collaborators in mid-2023 to fill this gap—to move beyond toy tasks and test performance on realistic long-form reasoning that practitioners actually care about.

How it works

The benchmark contains 4,693 examples across 21 datasets, each representing a realistic long-context task. For single-document QA, models read a 3-4K token document and answer factual questions (scored by exact match or F1). Summarization tasks measure ROUGE scores. Multi-document QA requires synthesizing information across several sources. Few-shot tasks give 5-10 examples in context and evaluate instruction-following accuracy. The scoring is task-dependent: exact match for factual QA, ROUGE for summaries, and pass@1 for code. Most tasks are evaluated on a 0-100 scale, aggregated across the eight domain categories.

What scores mean in practice

Human performance on LongBench averages around 85-90% (measured on a subset of tasks). As of late 2024, Claude 3.5 Sonnet scores ~75-80% and GPT-4o scores ~72-75%, while open models like Llama 2 70B score ~45-55%. Two years ago (late 2022), most open models couldn’t handle >2K tokens effectively; now the gap between commercial and open models has narrowed but remains significant on complex multi-document reasoning. The benchmark shows dramatic degradation on synthetic “needle” tasks for models with weak long-context implementations—some models that claim 128K window support drop to 30-40% accuracy when the answer is buried in the middle of the context.

Known limitations

• Synthetic task overweight: The needle-in-haystack and synthetic reasoning tasks don’t reflect real-world workflows and can be gamed by simple pattern-matching rather than genuine comprehension. Many models perform inconsistently on these vs. natural long documents.

• Dated dataset mix: Several datasets were released 2+ years before LongBench (2021-2022), raising contamination risks. Models trained on recent internet data may have seen some source documents or similar examples.

• Limited evaluation depth on reasoning: LongBench primarily measures retrieval and summary quality, not deep multi-step reasoning across long texts. It doesn’t test whether models can do novel inference that requires synthesizing information from disparate parts of a 10K-token document.

When to trust it (and when not to)

• Trust it for: Evaluating retrieval accuracy, summarization quality, and basic instruction-following on long documents. It’s useful for comparing commercial models or deciding whether a longer context window actually helps your use case.

• Don’t trust it for: Assessing reasoning quality, real-world multi-document synthesis, or predicting performance on genuinely novel long-context tasks. Benchmark scores correlate weakly with performance on custom, domain-specific long documents. Run your own test on real data before deploying a long-context model.