Ragas

Ragas is an evaluation framework for retrieval-augmented generation (RAG) systems built by AnúmIQ that measures quality without requiring reference answers.

The problem it set out to solve

RAG has become the default pattern for grounding LLMs in external knowledge, but evaluating whether a RAG pipeline actually works remained painfully manual. Teams were stuck either hiring humans to grade every output or relying on expensive LLM-as-judge comparisons that don’t scale. The builders saw teams shipping RAG systems into production with almost no visibility into whether retrieval was working, whether the LLM was hallucinating, or where the actual failures were happening.

The key insight

Most RAG evaluation frameworks require gold-standard reference answers — the thing you’re trying to avoid collecting in the first place. Ragas flipped the script: what if you could measure retrieval quality and generation fidelity using only the retriever’s output, the LLM’s response, and the original query? By decomposing RAG into its constituent parts (retrieval relevance, answer relevance, faithfulness, context precision), you can score each independently without human labels. This makes evaluation cheap enough to run continuously as part of your pipeline.

How it works (in plain terms)

Ragas breaks RAG evaluation into four main metrics. Faithfulness checks whether the LLM’s answer is actually grounded in the retrieved context by asking an LLM to identify claims and verify them against the documents. Answer Relevance measures whether the response actually addresses the question, independent of the retriever. Context Precision asks: of the documents you retrieved, how many were actually useful? And Context Recall estimates coverage—would the right answer have been in your retrieval results? The framework chains LLM calls strategically to compute these without needing any ground truth data. You get a score for each dimension, so you can see exactly where your pipeline is leaking.

What it looks like in practice

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevance,
    context_precision,
    context_recall,
)

results = evaluate(
    dataset=your_rag_dataset,  # questions, contexts, answers
    metrics=[
        faithfulness,
        answer_relevance,
        context_precision,
        context_recall,
    ],
    llm=ChatOpenAI(model="gpt-6-astra"),
    embeddings=embeddings_model,
)

print(results)  # see scores for each metric

Why it matters

• Makes RAG observable: You can finally see which component of your pipeline is failing—bad retrieval, bad generation, or both—without manual review at scale.
• Removes the evaluation bottleneck: No need to build or buy ground truth datasets. You can evaluate continuously as you iterate on retrievers, prompts, and LLMs.
• Changed how teams think about RAG quality: Shifted the conversation from “does this feel good?” to “here are four independent dimensions where we’re weak, let’s fix them.”

Where to go next

• Ragas GitHub — source, examples, and metric definitions
• Official docs — walkthrough of each metric and when to use them
• Measuring RAG performance paper — the research behind the metric design