How to Test a RAG Pipeline: A Practical Framework

A RAG (Retrieval-Augmented Generation) pipeline is tested by validating each of its four layers separately, retrieval, context assembly, generation, and grounding, against a golden dataset, rather than eyeballing end-to-end answers. Testing only the final answer hides where failures actually originate.

Most teams discover their RAG problems in production: the answer sounds right, cites nothing, and was built on the wrong retrieved chunk. This framework is how we make those failures visible before release.

Layer 1, Retrieval accuracy

Build a golden dataset of 50–200 real user questions, each mapped to the document chunks a correct answer requires. Then measure recall@k: of the chunks a correct answer requires, how many appear in your retriever's top k results? For example, if a question needs two policy chunks and only one shows up in the top five, recall@5 is 0.5 for that query.

• Recall@5 below ~0.8 means generation is guessing more than grounding
• Test paraphrased and misspelled variants of each question, retrieval that only works on exact phrasing fails real users
• Re-run the suite on every embedding model change and every reindex

Layer 2, Context assembly

Retrieved chunks pass through ranking, deduplication, and window packing before the model sees them. Assert that the top-ranked chunk actually appears in the final prompt, that near-duplicate chunks are collapsed, and that packing never truncates the one chunk that carried the answer.

Layer 3, Generation quality

With retrieval frozen (use a fixture of known-good context), evaluate the generator alone: does it answer from the provided context, decline when context is insufficient, and keep the required format? Scoring can combine exact-match checks for factual fields with an LLM judge for fluency, but the judge needs its own spot-check audit.

Layer 4, Grounding and citations

Every factual claim in the answer should be traceable to a retrieved chunk. Automate this by extracting claims and verifying each against the context, a claim with no supporting chunk is a hallucination, even if it happens to be true.

• Track hallucination rate per release, not per anecdote
• Fail the build when grounding drops below your baseline

Chunking: the silent variable under everything

Most RAG quality problems blamed on the model are chunking problems wearing a disguise. Chunk too small and answers lack context; too large and retrieval precision collapses because every chunk half-matches everything. Worse, naive splitting severs tables from their headers and answers from their questions. Treat chunking strategy as a versioned, tested artifact: when you change size, overlap, or the splitting logic, that's a reindex, and the golden-dataset suite runs before the change ships. Teams that skip this discover their 'model upgrade regression' was a chunk-boundary regression three weeks earlier.

Embedding swaps: the re-baseline ritual

New embedding models ship monthly and the benchmark deltas are seductive. The discipline: never swap on benchmarks. Re-embed a copy of the index, run the full retrieval suite against both, and compare recall@k per question category, general improvements routinely hide regressions on your domain vocabulary, the exact terms your users care about most. Keep the old index warm until the new one has a week of shadow traffic. This turns a risky migration into a boring one, which is the entire point of the framework.

Building the golden dataset (the unglamorous 80%)

The framework is only as good as the dataset under it. Source questions from real usage: support transcripts, search logs, sales calls, the phrasing users actually type, misspellings included. For each question, record which document chunks a correct answer requires (chunk IDs, not page numbers, they must survive reindexing) and what a correct answer must and must not claim.

Budget honestly: 50 questions is a strong start, 200 is a mature suite, and labeling takes a subject-matter expert about a day per fifty. Re-review quarterly, policies change, and a stale golden dataset fails silently in the opposite direction.

A recall@k check you can copy

import json, pytest
from app.rag import retrieve

GOLDEN = json.load(open("golden/rag_dataset.json"))

@pytest.mark.parametrize("case", GOLDEN, ids=lambda c: c["id"])
def test_recall_at_5(case):
 hits = {c.id for c in retrieve(case["question"], k=5)}
 required = set(case["required_chunk_ids"])

 recall = len(hits & required) / len(required)
 assert recall >= 0.8, (
 f"recall@5={recall:.2f} missing {required - hits}"
 )

The four layers at a glance

Make it a regression suite, not an audit

The framework only pays off when it runs on every prompt, model, or index change, exactly like a Playwright suite runs on every merge. That's the discipline gap between teams that ship AI confidently and teams that ship and hope.

FAQ: Should we choose onshore, offshore, or hybrid QA?

Structure matters more than geography. The five questions above filter equally in every time zone, an offshore team with named Senior review and honest reporting beats an onshore body shop every quarter. What time zones do change: overlap hours for stand-ups, and overnight Testing as a genuine advantage when feedback lands before your morning.

FAQ: What does a good first engagement look like?

Fixed scope, one artifact, two to four weeks: an audit with a written findings report, one stabilized suite area, or one evaluation harness stood up. You learn how the partner thinks, writes, and reports before committing to more, exactly the shape of our own free assessment, which is where most of our engagements begin.