How to Actually Evaluate Your RAG: Evaluation Metrics, Golden Datasets, and the Regression Gate

A RAG eval that grades the answer and skips the evidence lets a green dashboard sit happily on top of a retrieval layer that already failed. If the right context never arrived, the generation scores you are proud of are theater. The RAG evaluation metrics that show you where it broke measure the two halves apart: did retrieval bring back the right material, and then, separately, did the model use what it was handed.

Measuring those two halves apart is the part of RAG that rarely makes it into a demo. It first mattered to me the day dense-only search on our matcher buried an exact match and we rebuilt the retrieval as hybrid; I needed a way to judge any retrieval change across a whole set of queries.

The architecture piece in this series covers that hybrid search and the filtering. This one is how you know a retrieval change works, how you test it on every change, and how you keep it working as the corpus grows.

Two places it breaks, measured apart

A RAG system has two failure points, and they fail independently. Retrieval can fetch the wrong context, in which case the model never had a chance. Or retrieval can be perfect and the model still answers from its training, ignores what it was handed, or invents a detail that was never there. Score the system end to end with one number and you cannot tell which half broke, so you spend a day fixing the half that was fine. Measure the two sides separately, always.

The four core RAG evaluation metrics

Four metrics cover most of it. On the retrieval side, context precision asks how much of what you fetched was relevant, and context recall asks how much of what you needed you actually got. On the generation side, faithfulness asks whether every claim in the answer traces back to the retrieved context, and answer relevancy asks whether the answer addressed the question at all. Faithfulness is the one that catches a confident hallucination, so it is the one I read first.

One thing the retrieval pair comes down to in practice: did the chunk you needed land in the top handful you retrieved, because if it did not, no prompt and no larger model downstream can put it back. That failure often starts upstream, in how the document was chunked in the first place. Protect retrieval recall before anything else.

The four are where a team starts. Two more from Ragas earn their place on domain-heavy corpora: context entities recall, the share of the entities in the reference answer (names, numbers, codes) that survived into the retrieved context, and noise sensitivity, how often the system makes a wrong claim when handed relevant or irrelevant retrieved documents, where lower is better.

You can skip them on day one, but the day a legal or medical answer drops the one identifier that mattered, you will wish you had the metric that would have caught it.

Hit rate, recall@k and MRR for the retrieval half

Context precision and context recall are usually graded by a model reading the retrieved text; Ragas also ships string-matching and ID-based versions, and its context precision weights by rank, so a relevant chunk at the top of the window counts for more than one at the bottom. The retrieval half has plain ranking metrics as well: they run without a judge, straight off the labels in the golden dataset the next section builds, in milliseconds, so they are the ones to put in the test suite first.

Hit rate@k asks whether at least one of the chunks a question needs appears in the top k. Recall@k asks what fraction of all the chunks it needs made the top k, which is the one that matters for multi-document questions, where a hit on one of three sources still leaves the answer short.

MRR, mean reciprocal rank, takes the rank of the first relevant result for each question, inverts it (a top result scores 1, a third-place result scores a third) and averages across the set, so it rewards putting the right chunk first rather than somewhere in the window.

NDCG goes one step further and credits every relevant chunk by its position with graded relevance, which is worth the extra labeling only when your questions have degrees of right.

The golden dataset also carries questions the corpus cannot answer, on purpose, and for those rows the number is the abstention rate: the share where the system said it could not answer from the corpus. Track it next to recall, because the fixes that lift recall (looser filters, a bigger k) hand the generator more loosely related text, and together with a prompt that pushes it to answer, they can teach the system to answer questions it should refuse.

Pick k to match what the prompt actually sees. If the generator gets five chunks, recall@5 is the number; recall@20 is a diagnostic of what a reranker could rescue, which is a different question.

The golden dataset is the actual work

A golden dataset is a version-controlled file of real questions, each paired with the answer a domain expert accepts and the source passages that answer must come from, plus rows the corpus cannot answer. It is the fixed yardstick every retrieval and generation metric is scored against, so a change in the number means a change in the system.

Once you have that file the metrics are easy; building it is the job. Start with thirty to fifty questions as a starting heuristic and grow to a few hundred. At an 80% pass rate on 50 questions, the 95% interval is about plus or minus 11 points (Brown, Cai and DasGupta, 2001, on intervals for a proportion), and 200 questions bring it to about 5.5.

Four kinds of question earn their place. Plain factual ones with a single clear answer. Multi-document questions that force the system to stitch context from more than one place. Ambiguous ones with more than one fair reading. And the kind that is easiest to leave out, the question whose answer is nowhere in your corpus.

Building it, in order:

1. Pull questions from where users already ask them: support tickets, search logs, the chat transcript, the channel where people ask the expert. Kucia and Gawlik (2026) compared 1,851 model-generated questions against 322 authentic student queries; the real ones averaged 6.8 words, the synthetic ones 15.7, and tuning on the synthetic set picked a retriever that cost up to eight times the latency of the fastest configuration they tested. Write the questions the way users phrase them, typos included.

2. Label the required context by source document and location or text span, never by chunk id. Chunk ids change every time chunking changes, so a set keyed to them could never compare two chunking strategies; map spans to chunks at test time instead.

3. Allow more than one correct passage. Corpora repeat themselves, and a retriever that surfaces a near-copy of the labeled passage is right. The RocketQA authors (2020) checked the top-retrieved passages that MS MARCO had left unlabeled, and so treated as negatives, and found about 70% were positives or highly relevant. Record those near-copies in an accepted_alternates list under the passage they stand in for, so a correct hit is scored as one for hit rate and for recall alike.

4. Add the unanswerable rows, around one in ten, and write down why each has no answer, so the next person can re-check it after a big ingestion.

5. Have a domain expert review the expected answers, and review a sample of the whole set if a model wrote most of it. Rahmani, Craswell, Yilmaz, Mitra and Campos (SIGIR 2024) built a fully synthetic test collection and found it ranked retrieval systems in close to the same order as the human-judged one, a Kendall's tau of 0.86, while scoring systems of every type higher than real queries did.

A model-written set is good for comparing versions and catching regressions; treat its absolute pass mark as provisional until a person has reviewed a sample.

6. If a model writes the questions or grades the answers, use a different model from the one that answers them. Xu, Agrawal, Zouhar, Freitag and Deutsch (2025) ran Gemini 2.5 Pro, GPT-4.1 and Claude Opus 4 as test-set writers, graders and systems under test, and found that model-generated benchmarks favor the model that created them, with the test set and the evaluator each adding bias and the two compounding.

Their testbed was machine translation in low-resource languages, and on open-ended chat prompts the bias was much stronger for Bemba and Aymara than for English and Korean, so the risk grows where the model is least fluent.

Then keep it honest: rotate stale questions out and feed real production failures in, so the set describes the system you run rather than the one you shipped six months ago.

The question with no answer

The unanswerable question matters most, because it is the direct test of whether the system will say it does not know. Put a handful in the set and watch what they do. If their abstention rate holds while the answerable questions improve, you have a system that respects its own limits. If it answers them anyway, you have a liability that happens to demo beautifully.

A RAG system that answers every question with confidence is guessing with extra steps.

A golden dataset is just a file. Keep it boring and machine-readable, one record per line, labeled by source and span rather than chunk id.

{"id": "q-017", "type": "factual", "question": "Find a React Native engineer who has shipped a payments flow.", "expected": "profiles tagged react-native with a shipped payments integration", "required_context": [{"doc": "profiles/0142.md", "span": "Employment / 2023 / checkout and payments, React Native", "accepted_alternates": [{"doc": "profiles/0377.md", "span": "Projects / in-app payments, React Native"}]}]}
{"id": "q-052", "type": "multi_document", "question": "Who has both fintech and healthcare experience?", "expected": "the two profiles that list both verticals", "required_context": [{"doc": "profiles/0088.md", "span": "Employment, 2019-2024", "accepted_alternates": []}, {"doc": "profiles/0215.md", "span": "Employment, 2020-2025", "accepted_alternates": []}]}
{"id": "q-088", "type": "unanswerable", "question": "Find a certified Kubernetes administrator listed with a 2019 cert.", "expected": "abstain: the corpus lists that certification for nobody", "required_context": [], "why_unanswerable": "zero CKA holders in the corpus; re-check after each large ingestion"}

Keep it in version control, next to the code

A golden dataset belongs in the repository, beside the code it tests, because RAG testing is regression testing: the suite reruns whenever anyone changes the chunking, the embedding model, the fusion weight, the prompt, the generation model, the judge model or its rubric, and after any ingestion that grows the corpus by more than a few percent. A drop in recall@5 after a chunking change is then the chunking change, with the receipt in the diff, and that saves the day you would have spent debugging it.

Set the gate from a measurement: run the suite on the system you have, take each metric's pass rate, subtract its interval half-width, and raise it as the system earns it. A gate set at a round number someone remembered from a blog post fails in both directions, blocking honest noise or waving through a real regression. Treat every figure in this article, the thirty to fifty questions included, as a starting heuristic you replace with your own numbers after the first run.

One trap is that a judge or rubric change moves the baseline, so rerun the old system under the new judge and reset the gate before you grade the new one. Otherwise you credit or blame the change for what the judge did.

import json
from statistics import mean
from rag import retrieve, answer
from evals import grade, spans_to_chunk_ids

ROWS = [json.loads(line) for line in open("eval/golden_dataset.jsonl")]
GATE = json.load(open("eval/gate.json"))  # last accepted run minus its interval half-width
K = 5  # match what the generator sees

def retrieval_scores(row, hits):
    top = [h.chunk_id for h in hits]
    found, accepted = [], set()
    for item in row["required_context"]:
        ids = spans_to_chunk_ids([item, *item["accepted_alternates"]])  # mapped at test time, never stored
        found.append(any(c in ids for c in top))  # the passage or any near-copy of it counts
        accepted |= ids
    first = next((i + 1 for i, c in enumerate(top) if c in accepted), None)
    return any(found), sum(found) / len(found), (1 / first if first else 0.0)

def test_regression_gate():
    hit_, recall_, rr_, faithful_, answered_, abstained_ = [], [], [], [], [], []
    for row in ROWS:
        hits = retrieve(row["question"], k=K)
        out = answer(row["question"], hits)
        if row["type"] == "unanswerable":
            abstained_.append(out.abstained)
            continue
        hit, recall, rr = retrieval_scores(row, hits)
        hit_.append(hit); recall_.append(recall); rr_.append(rr)
        answered_.append(not out.abstained)
        faithful_.append(grade(out, hits, row).verdict == "pass")  # verdicts: pass, fail, unsure
    assert mean(hit_) >= GATE["hit_rate_at_5"]
    assert mean(recall_) >= GATE["recall_at_5"]
    assert mean(rr_) >= GATE["mrr_at_5"]
    assert mean(faithful_) >= GATE["faithfulness"]
    assert mean(answered_) >= GATE["answer_rate"]
    assert mean(abstained_) >= GATE["abstention_rate"]

The gate also holds the answer rate on the answerable rows. A system that abstains on everything makes no claim a grader can fault, so it would pass faithfulness and the abstention gate together while answering nothing.

Watch the unanswerable rows especially: as the corpus grows, a question that had no answer last quarter can quietly gain one, and an unanswerable test that is now answerable is silently grading the wrong thing. Re-verify that handful on every big ingestion.

How many questions before a gain is real

Compare two systems on the same questions, score every row pass or fail for both, and only the rows where they disagree carry information. The gap in pass rate is the net of those rows over the set size, so a 20-point gap on 50 questions is a net of ten questions.

If A wins ten and B wins none, a split that one-sided happens by chance about two times in a thousand, and the gap is real. If A wins twenty and B wins ten, the same 20-point gap, the chance of a split that lopsided is about one in ten, and you have learned almost nothing.

The floor is six: with fewer than six disagreements no gap of any size clears the usual bar, because six coin flips landing the same way happens about three times in a hundred and five happens six times in a hundred.

A single pass rate carries its own noise. At a pass rate near 80%, 30 questions give a 95% interval of about plus or minus 14 points and 50 give about 11 (Brown, Cai and DasGupta, 2001, on intervals for a proportion). A dashboard that moves from 78 to 84 on a 50-question set has moved by a net of three questions, inside that noise, which is the arithmetic behind growing the set to a few hundred.

Put that interval on every pass rate you publish. Questions drawn from one source document tend to pass and fail together, and Miller (2024) shows that an interval which treats them as independent comes out too narrow. So when you bootstrap the interval, resample by document rather than by question (Field and Welsh, 2007, on bootstrapping clustered data).

Keep a tuning change only when its paired gain holds on held-out questions. Qdrant's hybrid-search tuning guide frames it the same way: a bootstrap interval on per-query gain, a split between selection and held-out queries, and a configuration ships when the interval excludes zero and the gain holds on the held-out half.

The judge is useful, and fallible

You are not going to grade a few hundred answers by hand on every commit, so a language model scores most of them. Price the judge from your own numbers: tokens per verdict times the model's rate times the rows in the set, and the same multiplication for the production sample you grade each day. The figure belongs in the pull request next to the scores.

The judge also fails quietly. It tends to reward the better-formatted answer over the more correct one: Wu and Aji (2023) found answers with factual errors rated above answers that were too short or ungrammatical, so a tidy wrong answer can outscore a plain right one.

It favors its own outputs, which Panickssery, Bowman and Feng (2024) measured as self-preference, and text that reads like its own: Wataoka, Takahashi and Ri (2024) found judges rate lower-perplexity outputs higher than human raters do, whoever wrote them. And it gets unreliable on exactly the domain language your corpus is full of: Szymanski and colleagues (2024) found subject experts agreed with LLM judges 68% of the time in dietetics and 64% in mental health.

Two checks catch most of that. First, run the judge twice with the two candidates swapped, plus a repeat run on the same input, and count the verdict flips. Zheng and colleagues, in the MT-Bench paper (2023), found that most judges favor the first position, and their remedy is to call the judge in both orders and only declare a win when the same answer is preferred in both, calling it a tie otherwise.

Treat a flip on the repeat run as a tie too; a judge that flips often is telling you the rubric is loose.

Second, give the judge an explicit unsure outcome and score the judge itself against a domain expert. Anthropic's Demystifying evals for AI agents says to give a model grader a way out, an instruction to return Unknown when it lacks the information, and to calibrate it closely against human experts. Measure that calibration as two rates: how often it passes a truly correct answer and how often it fails a truly wrong one.

Those two rates decide how much the pass rate means. Lee, Zeng, Jeong, Sohn and Lee (2025) write the observed pass rate as the judge's sensitivity times the true rate plus its false-positive rate times the rest.

Work their formula with a judge that is right 80% of the time on both passing and failing answers: it reports 0.2 plus 0.6 times the true rate, so a system at a true 90% reads as 74%, a perfect system reads as 80%, and a real 10-point gain shows as 6. Their fix is to measure the two rates on a human-labeled calibration sample and solve back to the true rate, which is one more reason the expert review in step five is worth its afternoon.

Fix the grader before you trust the score in either direction. Anthropic reports a CORE-Bench run that moved from 42% to 95% after a researcher found rigid grading (a "96.12" marked wrong against an expected "96.124991" with more digits), ambiguous task specs and stochastic tasks that could not be reproduced exactly, and after the bugs were fixed and a less constrained scaffold was used. The grader was one cause among several, so record what else changed next to every score you report.

Anchor the judge with a rubric that scores facts rather than polish, run it on every commit, and have a person read a sample each month.

Tools that do most of this out of the box

Ragas implements the four metrics in the table plus context entities recall and noise sensitivity. DeepEval is an open-source framework that describes itself as similar to pytest but specialized for unit testing LLM apps, with faithfulness, answer relevancy, contextual precision, recall and relevancy, and synthetic dataset generation.

TruLens, open source and now maintained by Snowflake, scores the RAG triad of context relevance, groundedness and answer relevance with judges that explain their scores. Evidently, under Apache 2.0, evaluates, tests and monitors ML and LLM systems from classification to RAG, with LLM-as-a-judge descriptors and reports that turn into test suites with pass or fail conditions. Any of them will compute the numbers; the golden dataset, the gate and the judge checks are the part you still own.

FAQ

How do you evaluate a RAG system? Measure the two halves apart. Retrieval can fetch the wrong context, in which case the model never had a chance; or retrieval can be perfect and the model still ignores what it was handed. One end-to-end score cannot tell you which half broke. Score retrieval with hit rate@k, recall@k and MRR against a golden dataset, score generation with faithfulness and answer relevancy, and run both as a regression test on every change.

What are the core RAG evaluation metrics? Four cover most of it. Context precision, how much of what you fetched was relevant; context recall, how much of what you needed you actually got; faithfulness, whether every claim in the answer traces to the retrieved context; and answer relevancy, whether the answer addressed the question. Faithfulness catches the confident hallucination, so read it first. Hit rate@k, recall@k and MRR score the retrieval half without a judge, and the abstention rate covers the unanswerable rows.

What is a golden set in RAG evaluation? A golden set, or golden dataset, is a version-controlled file of real questions paired with expert-accepted answers and the source passages they must come from. Label the passages by document and text span rather than chunk id, record accepted alternates for near-copies, and include rows the corpus cannot answer. Draw the questions from real usage across four kinds: plain factual, multi-document, ambiguous, and unanswerable. Start with thirty to fifty questions as a starting heuristic and grow to a few hundred.

Why include questions with no answer? Because the unanswerable question is the direct test of whether the system will say it does not know, and the abstention rate on those rows is the number to watch. If it holds while answerable questions improve, the system respects its own limits. If it answers them anyway, you have a liability that happens to demo beautifully.

Where should the golden set live? In the repository, beside the code, because it is a test. The suite reruns on any change to chunking, the embedding model, the fusion weight, the prompt, the generation model, the judge or its rubric, and after a large ingestion. Set the gate from a measurement: run the suite on the system you have, take each metric's pass rate, subtract its interval half-width, and raise it as the system earns it. Re-verify the unanswerable handful on every big ingestion.

Can a language model grade the answers? Yes, and it should. Price the judge from your own numbers: tokens per verdict times the model's rate times the rows in the set, and the same multiplication for the production sample you grade each day. Judges tend to reward the better-formatted answer over the more correct one and can favor answers that resemble their own. Run the judge twice with the candidates swapped and count the flips, give it an unsure outcome, and score it against a domain expert.

What 2muchcoffee covers

We build production RAG systems and AI agents, grounded and evaluated. If your last retrieval change shipped on a feeling, our RAG engineers can put the loop in this article under it: the golden dataset, the gate, and the judge checks. The plain path to start is the AI work we do.

One concrete action

Pull twenty questions real users have asked your system, from logs, tickets or the chat channel, and keep them worded the way they were asked. For two of them, about one in ten as in step four, pick questions your corpus genuinely cannot answer. Run all twenty by hand and read the outputs. You will find the holes inside an hour, before you have built a single metric, and those twenty questions are the first rows of the golden dataset that keeps every later change honest.