The Benchmark — BLEU (Bilingual Evaluation Understudy)
A plain-English explainer of one AI evaluation benchmark: what it measures, how it works, and when to trust it.
BLEU (Bilingual Evaluation Understudy)
A metric that measures how closely machine-generated text matches human reference translations by counting matching word sequences.
What it measures
BLEU evaluates translation quality by comparing machine-translated output to professional human translations, looking for exact matches of individual words (unigrams) and sequences of 2, 3, and 4 consecutive words (bigrams, trigrams, 4-grams). It’s primarily used to benchmark machine translation systems, though it’s also applied to summarization and other text generation tasks where you have gold-standard reference texts to compare against.
Why it was created
BLEU was introduced by Kishore Papineni and colleagues at IBM in 2002 as a fast, automated alternative to labor-intensive human evaluation of translations. Before BLEU, evaluating machine translation required hiring professional linguists to manually assess each output—expensive and slow. BLEU promised a cheap, reproducible metric that could run in seconds.
How it works
BLEU calculates a score between 0 and 100 by computing the percentage of n-grams (word sequences) in the machine translation that appear somewhere in the reference translation(s). The algorithm weights n-grams: it counts matches for 1-word sequences, 2-word sequences, 3-word sequences, and 4-word sequences, then combines these with equal weight (25% each). A brevity penalty is applied if the output is shorter than the reference, preventing systems from gaming the score by producing very short translations. The final formula produces a geometric mean, resulting in a single number from 0–100.
What scores mean in practice
- Human baseline: Professional human translations score around 80–90 BLEU when compared to other professional translations of the same text.
- Current SOTA: Modern neural translation systems (like Google Translate, DeepL) score 40–60 BLEU on standard benchmarks like WMT (depending on language pair and domain). English-to-German might score 45 BLEU; English-to-Japanese might score 35.
- 2022 state-of-the-art: GPT-2 era systems typically scored 25–35 BLEU on similar tasks.
- Rule of thumb: 30+ is acceptable for rough translation; 50+ is approaching professional quality; 70+ is very high (usually only achievable with domain-specific data or very similar reference translations).
Known limitations
- Doesn’t capture meaning: BLEU only counts exact word matches. A translation can be semantically perfect but score 0 if it uses synonyms. “The quick brown fox” vs. “The fast brown fox” would receive no credit for the second word, even though the meaning is identical.
- Fragile to minor variations: Plurals, tenses, and word order variations penalty the score unfairly. A grammatically perfect translation using different word choice than the reference can score much lower than a clunky word-for-word translation that happens to match.
- Single-reference bias: BLEU improves dramatically with multiple reference translations, but many datasets only include one human translation. This makes scores artificially low and incomparable across benchmarks with different reference counts.
When to trust it (and when not to)
- Trust it: When you’re doing comparative benchmarking within the same benchmark (e.g., “Did our new model beat last month’s baseline?”) and you have multiple reference translations. BLEU is cheap, reproducible, and useful as a coarse signal for whether you’ve regressed.
- Don’t trust it alone: When evaluating actual translation quality for production use or when comparing across different benchmarks/language pairs. BLEU will mislead you—a perfectly usable translation might score 35, while a nonsensical translation with lucky word overlaps might score 40. Always supplement with human evaluation for consequential decisions.