Builders Spotlight — Ragas

Ragas

Framework for evaluating RAG pipelines with reference-free metrics, built by the Explodinggradients team.

The problem it set out to solve

RAG systems have become the default pattern for grounding LLMs in external knowledge, but evaluating them remains stuck in the old paradigm: comparing outputs to hand-crafted reference answers. That doesn’t work at scale. You can’t manually write gold-standard responses for thousands of retrieval scenarios, and even if you could, a “wrong” answer might actually be correct — just phrased differently. Teams were flying blind, unable to measure retrieval quality, generation relevance, or groundedness without expensive human review.

The key insight

You don’t need reference answers to measure RAG quality. Instead, you can use the LLM itself as a judge across multiple dimensions — does the retrieved context actually support the answer? Is the answer faithful to the retrieval? Did we retrieve relevant documents in the first place? By decomposing RAG evaluation into isolated, LLM-scored metrics (faithfulness, answer relevance, context precision, context recall), Ragas made evaluation practical and programmable. The philosophy is simple: lean on the model’s own reasoning ability to audit itself.

How it works (in plain terms)

Ragas breaks RAG evaluation into five core metrics, each answered by a different LLM call. For a given query and retrieved context, it asks: Did we retrieve the right documents? Are those documents relevant to the question? Does the generated answer actually come from the context (faithfulness)? Is the answer complete and relevant? Each metric produces a score between 0 and 1 without requiring a reference answer. The framework runs these evaluations in parallel, then aggregates them into a single pipeline score. You can swap evaluators (use GPT-4, local models, cheaper APIs) and adjust metric weights based on your use case.

What it looks like in practice

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision

dataset = [
    {
        "question": "What is RAG?",
        "answer": "Retrieval-Augmented Generation combines retrieval with generation.",
        "contexts": ["RAG is a technique that retrieves documents..."],
    }
]

results = evaluate(dataset, metrics=[
    faithfulness,
    answer_relevancy,
    context_precision,
])

print(results)  # Returns scores for each metric

Why it matters

• Closes the measurement gap: Teams can now run continuous evaluation on RAG pipelines without human annotation bottlenecks, making iteration fast and data-driven.
• Shifts debugging from guesswork to signals: When a pipeline fails, Ragas pinpoints whether the problem is retrieval quality, generation hallucination, or context ranking — not just “the answer was wrong.”
• Makes RAG engineering reproducible: Different retrieval strategies, chunking sizes, and rerankers can be compared quantitatively, turning RAG tuning into engineering rather than art.

Where to go next

• GitHub: explodinggradients/ragas
• Docs: ragas.io — extensive guides on metrics, custom evaluators, and integration with evaluation frameworks
• Talk: Exploding Gradients team has shared talks on RAG evaluation patterns at AI conferences; check their blog for metric methodology deep-dives