The Benchmark — RULER (Recursive Length-Extended Reasoning)
A plain-English explainer of one AI evaluation benchmark: what it measures, how it works, and when to trust it.
RULER (Recursive Length-Extended Reasoning)
RULER measures how well language models can find and reason over relevant information in very long contexts (up to 4,096 tokens and beyond).
What it measures
RULER tests a model’s ability to locate specific facts or patterns buried in long documents and use them to answer questions—a capability called “needle-in-haystack” retrieval combined with reasoning. Rather than just checking if a model can find information, RULER requires models to retrieve multiple pieces of information and synthesize them to answer a question, simulating real-world document analysis tasks like contract review, research synthesis, or multi-document question-answering.
Why it was created
Long-context windows became a major selling point for new models around 2023-2024 (Claude 3.5 with 200k tokens, GPT-4 Turbo with 128k), but there was no standardized benchmark to measure whether models actually use that capacity effectively. Earlier “needle in a haystack” tests were too simplistic—they only required finding one isolated fact. RULER, introduced by researchers at MIT-IBM Watson AI Lab and others in late 2024, addresses this gap by testing retrieval and reasoning together across genuinely long documents.
How it works
RULER generates synthetic documents (up to 4,096 tokens long) with structured question-answer pairs embedded at various positions. Models must locate multiple relevant facts scattered throughout the document and combine them to answer complex questions. Scoring is exact-match accuracy: either the model’s answer matches the correct answer or it doesn’t. The benchmark includes variants at different context lengths (512, 1,024, 2,048, 4,096+ tokens) and difficulty levels—some questions require retrieving information from one location, others require synthesizing information from multiple sections. Results are reported as accuracy percentages at each context length.
What scores mean in practice
Human performance on RULER is typically 95%+ accuracy since humans are good at skimming documents. As of January 2025, Claude 3.5 Sonnet achieves ~92-95% on mid-length contexts (2,048 tokens) but drops to ~85-90% at maximum length. GPT-4 Turbo scores slightly lower (~88-92% at 2,048 tokens). Smaller models like Llama 2 70B collapse rapidly past 2,048 tokens, dropping to 60-70% accuracy. For reference, a completely random guesser on multiple-choice variants would score ~25%. The key signal is degradation rate—how much accuracy drops as context length increases. A 5-10% drop from 2k to 4k tokens is strong; a 20%+ drop suggests the model struggles with genuine long-context reasoning.
Known limitations
- Synthetic data bias: RULER uses artificially constructed documents with regularized structure, not real messy text. Models may perform differently on actual long documents with natural writing, redundancy, and ambiguity.
- Limited reasoning depth: While RULER requires multi-fact retrieval, the reasoning is still relatively shallow—most answers are extractive or require simple arithmetic. Deep multi-step reasoning across dozens of pages isn’t tested.
- No contamination controls: These tests are new enough that model training data overlap is unlikely, but the synthetic nature means contamination is hard to detect if it does occur.
When to trust it (and when not to)
- Trust it for: Comparing long-context capabilities between models and understanding degradation patterns. If a model claims 200k-token support, RULER is a reasonable sanity check on whether that actually works for retrieval tasks.
- Don’t trust it alone for: Predicting real-world performance on contract review, legal research, or other practical long-document tasks. Performance on synthetic, well-structured documents often doesn’t transfer to messy real-world data. Pair it with task-specific evaluation or user testing before deploying.