How to evaluate RAG systems

A retrieval-augmented generation system can return a polished answer and still fail in several different ways. The retriever may miss the relevant document, rank an outdated passage first, send too much context to the model, or retrieve the right evidence only for the generator to ignore it.

RAG evaluation should identify which part failed. That requires measuring retrieval, generation, and the complete user outcome separately before connecting them into one release decision.

What is RAG evaluation?

RAG evaluation is the process of measuring how effectively a retrieval-augmented generation system finds relevant evidence and uses it to produce a correct, grounded, useful response.

A complete evaluation answers three groups of questions:

1. Retrieval: Did the system find the evidence needed to answer the question?
2. Generation: Did the model use the supplied evidence faithfully and produce a useful response?
3. End-to-end outcome: Did the system solve the user's task within the required safety, latency, and cost boundaries?

This separation is essential for diagnosis. If retrieval never surfaced the refund-policy exception, changing the final prompt may not help. If the correct policy was present but the answer contradicted it, the generator or its instructions need attention.

The RAGAS research framework helped formalize this decomposition by evaluating the relevance and focus of retrieved context, faithfulness to that context, and response quality. A production evaluation program can use those ideas without depending on one framework or one aggregate score.

Define the RAG task before selecting metrics

Start with the application outcome and consequence of failure.

A documentation assistant may need to answer accurately, cite the exact page, and admit when the documentation does not contain the answer. A compliance assistant may need to retrieve only approved policies valid for a jurisdiction and effective date. A support agent may use retrieval as one step before taking an action.

Document:

• The users and question types in scope.
• The sources the system is allowed to use.
• Freshness and permission requirements.
• Whether an answer requires citations.
• When the system should abstain or escalate.
• The acceptable latency and cost per task.
• The failures that must block a release.

These decisions determine the dataset, labels, metrics, and evaluation methods. A generic benchmark cannot define product-specific correctness.

Evaluate the retrieval layer

Retrieval evaluation measures whether the system supplies useful evidence to the generator.

Context recall

Context recall asks whether the retrieved set contains the evidence required to answer the question. When the evaluation dataset includes known relevant documents or passages, calculate how much of that evidence appears in the top results.

Low recall can indicate poor chunking, incomplete indexing, weak query formulation, inappropriate filters, embedding mismatch, or a top-k value that is too small.

Context precision

Context precision asks how much of the retrieved material is relevant. Returning the correct passage alongside many irrelevant chunks may preserve recall while increasing tokens and distracting the generator.

Inspect precision at the ranks the application actually sends to the model. Relevant evidence at rank 20 does not help if only the first five chunks enter the prompt.

Ranking quality

Metrics such as mean reciprocal rank, normalized discounted cumulative gain, hit rate, or precision at k can evaluate where relevant items appear. Select the measure that reflects user experience and the downstream context limit.

Ranking labels may be binary or graded. A current policy page can be highly relevant, a related FAQ partially relevant, and an obsolete policy actively harmful.

Source quality and freshness

Retrieval can be semantically relevant and still unsafe to use. Evaluate whether results satisfy permissions, tenant boundaries, effective dates, source authority, language, and document status.

Treat unauthorized or superseded context as a hard failure where appropriate, not merely a lower relevance score.

Context diversity and redundancy

Several nearly identical chunks can crowd out complementary evidence. Measure duplicate content, source diversity, and whether the retrieved set covers the different facts required for a multi-part answer.

Evaluate the generation layer

Generation evaluation assumes the exact retrieved context is available for inspection. Preserve it with the response so evaluators can determine how the model used it.

Groundedness or faithfulness

Groundedness checks whether factual claims in the response are supported by the supplied context. A grounded answer may still be incomplete or based on a poor source, which is why the retrieval checks remain separate.

Evaluate individual claims when the risk is high. A single response-level score can hide one unsupported statement inside an otherwise accurate answer.

Answer correctness

When a reference answer or verifiable system state exists, check whether the response reaches the correct conclusion. Use deterministic checks for structured values and task-specific rules; use human or model-based review for nuanced language.

Correctness and groundedness are related but different. A response can make a correct statement that is unsupported by the retrieved context, or faithfully repeat an incorrect source.

Answer relevance and completeness

Measure whether the response directly addresses the question and includes the information needed to complete the task. Penalize unnecessary background when it obscures the answer or increases cost.

For multi-part requests, score each requirement separately. This makes missing elements actionable.

Citation quality

If the product shows citations, test whether each citation exists, supports the associated claim, points to the intended source, and resolves for the user. Citation presence alone is not evidence of citation correctness.

Abstention behavior

Include unanswerable and insufficient-context cases. Evaluate whether the system recognizes missing evidence, communicates uncertainty, asks for clarification, or escalates instead of inventing an answer.

Evaluate the complete RAG experience

End-to-end evaluation connects retrieval and generation with user and operating outcomes.

An end-to-end score is useful for comparing releases, but keep the component scores available. Otherwise, a quality improvement may hide a retrieval regression or a latency increase.

Build a representative RAG evaluation dataset

A useful dataset contains more than questions and ideal answers. Each case should record:

• The user question and relevant conversation history.
• The expected answer or outcome.
• Required and acceptable source documents.
• Explicitly irrelevant, outdated, or prohibited sources when important.
• The reason the case matters and its risk category.
• Language, user segment, permissions, and effective date.
• Expected abstention or escalation behavior.

Include ordinary questions, multi-hop questions, ambiguous wording, exact identifiers, misspellings, conflicting documents, recently updated sources, permission boundaries, and questions the corpus cannot answer.

Start with a small set reviewed by domain experts. Expand it with sanitized production questions, low-rated conversations, search reformulations, failed citations, and incidents. Version the dataset and its source corpus so results remain reproducible.

Choose complementary evaluation methods

Deterministic retrieval metrics

Use known relevant documents and ranking metrics where ground truth is available. These checks are inexpensive and easy to compare across index, embedding, chunking, and reranking changes.

Rule-based response checks

Validate citation structure, required fields, prohibited phrases, source URLs, JSON schemas, and exact values with code. Do not use a model-based judge for properties that software can verify reliably.

Model-based evaluators

Use a clearly defined rubric for groundedness, answer relevance, completeness, and qualitative citation support. Supply the evaluator with the question, retrieved context, response, and reference evidence needed for the decision.

Calibrate every model-based evaluator against expert-reviewed cases and track its errors by task type. The evaluator is another versioned model dependency, not ground truth.

Human review

Use domain experts for high-risk cases, ambiguous labels, new failure types, and evaluator calibration. Structured rubrics make the resulting judgments useful for automation and trend analysis.

A practical RAG evaluation workflow

1. Define the outcome and risk boundaries. Decide what must be correct, cited, current, permitted, and fast.
2. Create a versioned seed dataset. Include expected evidence and abstention cases, not only ideal answers.
3. Evaluate retrieval independently. Compare chunking, embeddings, filters, query rewriting, top-k, and reranking.
4. Freeze retrieved context when comparing generators. This prevents retrieval changes from confounding model or prompt comparisons.
5. Evaluate generation by dimension. Measure groundedness, correctness, relevance, completeness, citation quality, and abstention.
6. Run end-to-end checks. Include task success, policy, latency, and cost.
7. Inspect failures by segment. Compare performance by task, source, language, permissions, and question complexity.
8. Gate releases. Set hard requirements for safety and source policy, plus acceptable regression budgets for quality and performance.
9. Evaluate production samples. Use traces and user feedback to discover cases the offline set missed.
10. Promote failures into regression tests. Preserve the question, corpus or source version, retrieved context, and expected behavior.

Diagnose RAG failures systematically

Use the observed pattern to narrow the cause.

Preserve the trace behind each failure. It should connect the query, retrieval configuration, ranked documents, prompt, model, response, evaluator results, latency, cost, and application version.

Common RAG evaluation mistakes

Scoring only the final answer

This makes retrieval and generation failures indistinguishable. Evaluate the ranked context and how the response uses it.

Using synthetic questions as the entire dataset

Synthetic data can expand coverage, but generated questions often reflect the corpus structure rather than real user intent. Combine them with expert cases and production evidence.

Treating one model-based score as ground truth

Automated evaluators can be biased or inconsistent. Calibrate them, preserve explanations, and retain deterministic and human checks.

Ignoring corpus versions

An evaluation cannot be reproduced if the index changed silently. Version the source corpus, chunking, embedding model, and retrieval configuration.

Optimizing retrieval metrics without the user outcome

Higher recall may increase context length and reduce answer quality or latency. Evaluate the full system after tuning a component.

Polyaxon and the RAG-evaluation roadmap

Polyaxon's current tracking and orchestration capabilities provide a foundation for versioning evaluation datasets, retrieval configurations, prompts, models, metrics, and automated workflows. Teams can compare runs and connect evaluation evidence to the artifacts and configurations that produced it.

Full LLM and application evaluation is an active area on the Polyaxon roadmap, alongside OpenTelemetry-compatible observability, an AI gateway, prompt management, and agent sandboxing. The direction is to connect a RAG trace with retrieval evidence, evaluator results, production feedback, and the exact application version.

Evaluate the evidence path

RAG quality depends on a chain: the corpus contains the right knowledge, retrieval finds it, ranking prioritizes it, the model uses it faithfully, and the final response solves the user's task.

Measure every link in that chain. A decomposed evaluation tells the team what to improve, while the end-to-end evaluation confirms that the change actually benefits users. Continue with LLM-as-a-judge for semantic scoring and offline versus online evaluation for the release feedback loop.