Lab 13

Evaluating RAG Systems: Retrieval, Faithfulness and Failure Analysis

Evidence recall Faithfulness Citations Ablations

Work in progress These notes are still being written and reviewed: content may change, and some parts may still contain mistakes.

Overview of the lab

Lab 12 built the RAG pipeline (documents → chunk → index → retrieve → rerank → select context → prompt → LLM → answer). This lab asks: how do we know whether the RAG system is actually good? A single end-to-end accuracy number is not enough: the answer may not be in the corpus, the chunking may be poor, the retriever may miss the evidence, the reranker may remove it, the prompt may hold irrelevant context, the model may ignore the evidence, the answer may be unsupported, the citation may be wrong.

Evaluate the pipeline by stage, not only by final answer quality.

Why RAG evaluation is difficult: three layers.
Retrieval-side evaluation: evidence recall, context precision, MRR, nDCG.
Generation-side evaluation: answer correctness and faithfulness.
Citation evaluation: citation precision and recall, evidence coverage.
Building a RAG evaluation dataset, with unanswerable questions.
Comparing two RAG configurations, on a small collection.
Failure analysis: labelling what went wrong.
Ablation studies and LLM-as-judge.
The whole course in one pipeline, the summary and optional exercises.

Learning objectives

By the end you should be able to explain:

why RAG evaluation is a multi-stage problem, and how retrieval metrics from classical IR still apply;
the difference between evidence recall and context precision;
what answer correctness and faithfulness / groundedness measure, and how citation quality differs from answer correctness;
why end-to-end answer accuracy can hide retrieval failures, and why retrieval success does not guarantee generation success;
how to design a small RAG evaluation set, what an ablation study is, and how to diagnose common RAG failure modes;
why automated LLM-based evaluators must themselves be treated as imperfect measurement tools.

Part I

Why RAG evaluation is difficult

Suppose a system gives the wrong answer. Why? Walk down the questions; the first "no" names the failing stage (the debugging tree of Lab 12). A final answer metric alone cannot tell us which component failed.

Was the information present in the corpus?no → corpus / ingestion failure

Was the right chunk created?no → chunking failure

Was it retrieved?no → retrieval failure

Was it retained after reranking / context selection?no → reranking / context-selection failure

Did the model use it correctly?no → generation failure

Was the citation correct?no → citation failure

✅ success

At least three layers

Layer	Question
1. Retrieval quality	Did we retrieve the right evidence?
2. Generation quality	Did the model answer correctly and stay grounded in the evidence?
3. End-to-end quality	Did the whole system satisfy the user's information need?

These should not be collapsed too early. Question: How many ECTS credits is the master's thesis worth? The evidence exists: "The master's thesis is worth 30 ECTS credits." Three systems:

System	Correct passage retrieved?	Answer	Correct?	Grounded?
A	yes	The thesis is worth 30 ECTS.	✅	✅
B	yes	The thesis is worth 20 ECTS.	❌	❌
C	no	The thesis is worth 30 ECTS.	✅	❌

Looking only at answer correctness, A and C are both right. But ❗ C is dangerous: it produced the correct answer without retrieved support (from the model's memory, or by luck). RAG evaluation must separate correctness from grounding.

Part II

Retrieval-side evaluation

Everything from Lab 05 still applies. For a query, suppose we know which chunks contain the required evidence: gold evidence C4, C11; the retriever returns C2, C11, C7, C19, C4. Both supporting chunks are within the top 5.

📐 Evidence Recall@k and Context Precision@k

\text{EvidenceRecall@}k = \frac{\text{gold evidence chunks retrieved in the top } k}{\text{all gold evidence chunks}} \qquad \text{ContextPrecision@}k = \frac{\text{relevant retrieved chunks}}{k}

Evidence recall: did the evidence reach the context? Context precision: how much of the context is actually useful?

Try it: the outline's example

The ranked list, continued past rank 5 with three more chunks (invented). Move \(k\).

\(k\)5

Why Recall@k matters for RAG: the generator only sees the selected context. If the necessary evidence is outside the context window, generation cannot reliably use it. Retrieval recall places an upper bound on what the downstream system can do.
Recall and precision pull in different directions: increasing \(k\) may improve evidence recall but may reduce context precision. A familiar IR trade-off (Lab 12's top-k widget shows it on Cranfield).
MRR and nDCG still matter: if one supporting passage is sufficient, MRR asks how early the first supporting chunk appears; if evidence has graded usefulness (primary, secondary, weakly related), nDCG is meaningful. RAG does not invalidate classical IR metrics; it reuses them.

Part III

Generation-side evaluation

Once the correct evidence is in the prompt, we still need to evaluate the generated answer. Two central dimensions, not the same thing: answer correctness and faithfulness.

Answer correctness

Gold answer 30 ECTS; generated The master's thesis is worth 30 ECTS credits. Correct. A generated answer can be compared with a reference answer, a known fact, human judgment, a task-specific exact match or semantic equivalence, depending on the task.

Exact match: for short factual QA (gold 30, prediction 30) it can work, but it is brittle: "The thesis carries 30 European Credit Transfer System credits" is semantically correct, but not string-identical.
Token-level overlap (precision, recall, F1 over answer tokens) can be useful for short extractive answers, but it is weak for open-ended generation.
Semantic answer evaluation: does this answer mean the same thing as the reference? This may need human evaluation, task-specific rules, semantic similarity or LLM-based judging; each has limitations.

Try it: exact match and token F1

SQuAD-style scoring against the gold answer: lowercase, drop punctuation and the articles a, an, the, split on spaces; F1 over the shared tokens. Pick a prediction, or type your own.

gold

prediction

Faithfulness / groundedness

Faithfulness asks: are the claims in the answer supported by the retrieved context? Break the answer into claims, ask for each "supported by the context? yes / no", and

\text{Faithfulness} = \frac{\text{supported claims}}{\text{all claims}}

Try it: judge the claims

Context: The programme requires 120 ECTS. The thesis is worth 30 ECTS. Click each claim of the answer to mark it supported (✔) or not (✘), then check your labels.

Partially unfaithful: an answer may contain correct statements and still add an unsupported one (the internship).
❗ Correct but ungrounded: factually correct, but not supported by the retrieved evidence. For RAG this should not automatically count as full success: the purpose of retrieval augmentation is to provide evidence.
❗ Grounded but wrong: supported by the context, but the context itself is wrong or outdated (an answer from the 2024 fee table in 2026). Another layer: source quality. RAG can only be as trustworthy as its corpus.

Part IV

Citation evaluation

"The thesis is worth 30 ECTS [2]." The citation must be evaluated separately from the answer.

📐 Citation precision and recall

\text{CitationPrecision} = \frac{\text{correct supporting citations}}{\text{all citations produced}} \qquad \text{CitationRecall} = \frac{\text{claims with valid supporting citations}}{\text{claims requiring support}}

Precision: of the citations produced, how many support their claim? Recall: of the claims that need support, how many got a valid citation?

Try it: check the citations

Three passages, an answer with three claims, and the citation the model attached to each (invented). Change the citations and watch both scores.

Evidence coverage. Some answers require several pieces of evidence. Can a student graduate if the thesis is incomplete? needs one chunk describing the total credits and one describing the thesis requirements; a retriever may find only one. Some questions require multi-evidence retrieval (Q5 and Q8 of the evaluation set below).

First-half recap

Layer	Example metric
Retrieval	Evidence Recall@k
Context quality	Context Precision@k
Ranking	MRR / nDCG
Answer quality	correctness
Grounding	faithfulness
Source use	citation precision / recall
End-to-end	task success

To evaluate any of this, we need an evaluation set.

Part V

Building a RAG evaluation dataset

A useful RAG evaluation set contains more than questions. For each example store the question, the reference answer, the supporting document(s) and supporting chunk(s); optionally the answer type, difficulty, topic, whether it requires multiple chunks, whether the system should abstain.

eval_set.py Python

examples = [
    {
        "id": "Q1",
        "question": "How many ECTS credits is the master's thesis worth?",
        "reference_answer": "30 ECTS",
        "supporting_chunks": ["D2_C1"],
        "answerable": True,
        "category": "fact_lookup",
    },
    {
        "id": "Q11",
        "question": "What is the tuition fee for 2030?",
        "reference_answer": None,
        "supporting_chunks": [],
        "answerable": False,
        "category": "unanswerable",
    },
]

❗ Include unanswerable questions. For What is the tuition fee for 2030?, when the corpus has no such information, the correct behavior is "insufficient information", not a guess. Without such questions, a model that always answers confidently may look good. A robust RAG system should know when the evidence is missing: abstention quality.
Diversity: exact fact lookup, paraphrase, rare terminology, multi-hop / multi-evidence, ambiguous queries, unanswerable queries. It makes failure analysis more informative.
Keep evaluation data separate: if you tune chunk size, top-k, RRF parameters, the reranker or the prompt on the examples used for final reporting, you risk overfitting the evaluation set. Development set → tune; test set → final evaluation. The same principle as in classical IR.

See it: the evaluation set of this lab

Ten short handbook documents (842 words, invented, in the style of Lab 12's regulation) and twelve questions, covering every category above. The gold evidence is stored as sentences, so that it can be matched against any chunking: a chunk supports a question if it contains a whole evidence sentence.

Part VI

Comparing two RAG configurations

The outline's two systems: A = chunk size 150, top-k 3, dense retrieval; B = chunk size 400, top-k 8, hybrid retrieval + reranker. Our documents are short (64–123 words), so the chunk sizes are scaled down: A uses 60-token chunks, B whole documents, with top 5 (8 of 10 documents would be almost the whole collection). Do not assume B is better merely because it is more complex: measure it.

retrieval_metrics.py Python

def evidence_recall_at_k(retrieved_ids, gold_ids, k):
    retrieved = set(retrieved_ids[:k])
    gold = set(gold_ids)
    if not gold:
        return None                  # unanswerable: no evidence to recall
    return len(retrieved & gold) / len(gold)

def context_precision_at_k(retrieved_ids, gold_ids, k):
    retrieved = retrieved_ids[:k]
    if not retrieved:
        return 0.0
    gold = set(gold_ids)
    useful = sum(chunk_id in gold for chunk_id in retrieved)
    return useful / len(retrieved)

Try it: System A vs System B

Real rankings (all-MiniLM-L6-v2, rank_bm25, RRF, the ms-marco cross-encoder) over the ten answerable questions. Change any setting of either system.

End-to-end inspection, question by question

💡 Evidence recall here counts evidence sentences held whole by a top-k chunk; context precision the share of the top \(k\) that holds at least one. In the default comparison B finds all the evidence (1.00 against 0.80) but hands the model about three times as many tokens; neither is "better" without looking at the downstream answers. Try 30-token chunks: the retrievers barely matter, because 5 of the 13 evidence sentences are cut in two and no chunk holds them whole.

Why both metrics? A system with Recall@5 = 1.0 and Context Precision@5 = 0.2 found all the evidence but added a lot of noise; one with Recall@5 = 0.5 and Context Precision@5 = 1.0 has a clean context but missed half the evidence. Neither is obviously superior without considering the downstream task.
End-to-end answer inspection: for several questions print the question, the retrieved chunks, the generated answer, the reference answer and the citations. One of the most valuable debugging views in a RAG system. Do not evaluate only a spreadsheet of aggregate scores. (The inspection above stops at the context: no generator runs on this page.)

Part VII

Failure analysis

For every failed example, assign a failure category:

CORPUS_MISSINGBAD_CHUNKINGRETRIEVAL_MISSRERANKING_MISSCONTEXT_NOISEGENERATION_ERRORUNFAITHFUL_CLAIMBAD_CITATIONFAILED_TO_ABSTAIN

Query	Final result	Failure category	Evidence
Q1	wrong answer	`RETRIEVAL_MISS`	gold chunk ranked 12
Q2	wrong answer	`GENERATION_ERROR`	gold chunk at rank 1
Q3	unsupported claim	`UNFAITHFUL_CLAIM`	claim absent from the context
Q4	wrong citation	`BAD_CITATION`	cited chunk unrelated

This turns vague "RAG quality" into something diagnosable.

Retrieval failure: gold chunk at rank 17, context cutoff top 5. No prompt engineering can fix this query: the retriever must improve.
Generation failure: gold chunk at rank 1, correct evidence in the prompt, wrong answer. Retrieval succeeded; changing the vector database will not solve the real problem.
Chunking failure: "The master's thesis is worth | 30 ECTS credits." cut between chunk A and chunk B: neither holds a complete answer. The retriever may be working correctly over badly constructed units.
Corpus failure: if the answer is absent from the indexed corpus, retrieval cannot succeed. Before changing models, inspect the data.

See it: the retrieval-side failure distribution

For System A and System B of Part VI, every question labelled automatically up to the context: the labels that need a generated answer (generation, faithfulness, citation, abstention) cannot be assigned without one.

💡 The two unanswerable questions are CORPUS_MISSING by design: the right outcome is an abstention, and whether the system abstains (or earns FAILED_TO_ABSTAIN) is decided by the generator, since the retriever returns chunks anyway. Decide which subsystem deserves attention first from the counts, not from impressions.

Part VIII

Ablation studies

An ablation removes or changes one component while keeping the rest as fixed as possible. It answers: which component actually contributed? The outline's four retrieval stacks on Cranfield (the runs of Lab 11, reranking the top 50, 225 queries):

💡 Adding the reranker to dense retrieval helps on every measure; adding it to the hybrid lowers all four. The most complex stack is not the best one on Cranfield. On a real RAG system you would add answer correctness and faithfulness columns, which need a generator and judgments.

Chunking ablation: compare 100-, 300-, 700-token chunks with the retriever fixed; if performance changes, chunking is contributing. On the small collection, dense retrieval with top 3:
Top-k ablation: \(k\) = 1, 3, 5, 10, measuring evidence recall, context precision and answer quality. It reveals whether more context helps or hurts (Lab 12's top-k widget).

Part IX

LLM-as-judge

For open-ended answers human evaluation is expensive. One approach: give another language model the question, the reference answer, the retrieved context and the generated answer, and ask it: Is the answer correct? Is every claim supported? This can be useful at scale, but ❗ it is still a measurement model, and it can make mistakes:

sensitivity to prompt wording; inconsistent judgments;
preference for longer answers; model-specific bias;
difficulty with subtle factual errors and with domain-specific evidence.

Do not treat an LLM judge as ground truth. For important experiments: manual spot checks + automated evaluation. The same holds for RAG-evaluation frameworks: introduce them only once you understand what the metrics are trying to measure, and do not let a framework replace the evaluation model.

Part X

The whole course in one pipeline

documents, indexing lexical retrieval evaluation web / link structure query expansion neural representations sentence embeddings, semantic search vector index / vector database, dense retrieval hybrid retrieval, reranking RAG RAG evaluation

Summary: key takeaways

RAG must be evaluated by stage, not only by final answer quality.
Evidence Recall@k tells us whether the required evidence reached the context window.
Answer correctness and faithfulness are different dimensions.
Citation quality should be checked separately from answer quality.
Ablation and failure analysis are more informative than blindly changing models.

Optional exercises

A small RAG evaluation set. 10–20 questions with reference answers and gold supporting chunks, plus 2–5 unanswerable questions (a JSON list like eval_set.py is enough). Evaluate two configurations (e.g. dense retrieval with top 3 against hybrid + reranking with top 5) and report Evidence Recall@5, Context Precision@5, answer correctness, faithfulness, citation precision and abstention accuracy. Then inspect at least five failed examples by hand.
Ablation. Choose one variable (chunk size, overlap, retrieval method, top-k, reranking) and change only that. Report retrieval metrics, generation metrics and latency if available. What changed, and at which stage of the pipeline did the effect appear?
Error analysis. Label every failed question with one of the nine categories, count them in a failure distribution, and decide which subsystem deserves attention first. Tools: numpy, pandas, and ir_measures for the retrieval side.