Data Quality for RAG: Your Retrieval Is Only as Good as the Pipeline Behind It

When a retrieval-augmented system answers badly, the investigation almost always starts at the model. The prompt gets rewritten, the chunk size gets tuned, someone proposes a different embedding model, and an evaluation harness is pointed at the whole thing to prove the change helped. Sometimes it does.

Often the defect was three steps upstream, in the pipeline that assembled the corpus. The document that would have answered the question was never ingested because a nightly job failed quietly and retried into a partial load. Or it was ingested twice, and the duplicate outranked the canonical copy. Or a schema change renamed the column carrying the effective date, so every chunk still says it is current. None of that is a model problem, and no amount of prompt engineering fixes it.

The four failure modes that look like model problems

What these share is that the system keeps working. There is no exception, no failed build and no alert, because none of them is an error in the code that ran. They are errors in the data it produced, and only something that inspects the data will find them.

• Silent partial loads. An ingestion job fails midway, retries, and lands a fraction of the source. Retrieval still returns the top-k most similar chunks, so nothing looks broken, the answers are simply worse than they should be.
• Duplicate documents. The same policy exists twice under different keys. Both are retrieved, the model sees two near-identical passages, and treats the agreement as corroboration. Confidence goes up while accuracy does not.
• Stale rows with fresh timestamps. A transformation writes the load time rather than the source's effective date. Every document now claims to be current, and recency filters become worse than useless because they actively prefer the wrong rows.
• Truncation at a column boundary. A varchar limit clips the end of a long document. The chunk still embeds, still retrieves, and still reads as complete, because a truncated paragraph rarely announces itself.

Test the corpus before you test the model

The checks are unglamorous and mostly predate AI by decades. Reconcile the document count in the vector store against the source of record, not against the last run. Assert uniqueness on whatever should be unique, and count near-duplicates as well as exact ones. Compare the distribution of document dates against what the source actually contains, which catches the timestamp-overwrite case immediately. Check character length at the tail of the distribution, where truncation lives.

Run them in CI on every ingestion, and fail the pipeline rather than reporting into a dashboard nobody opens. If that sounds like ordinary ETL and data warehouse Testing, that is because it is. The retrieval layer is new, the discipline is not.

The evaluation set has the same problem

There is a second, quieter version of this. Golden datasets are frequently assembled by querying the same warehouse the application reads from, which means an evaluation set inherits every defect in those tables and then hardens them into ground truth. The model is now scored against the errors, and a change that fixes real behaviour can measure as a regression.

Treat the golden dataset as a deliverable with its own acceptance criteria, validated at build time and versioned alongside the harness. Our AI Quality Engineering practice keeps the eval set and the corpus under the same reconciliation checks, precisely because they fail the same way.

Where to start if you own one of these systems

If the answers are uncomfortable, that is useful information and it is cheaper to have now. We run this as a scoped assessment and hand back written findings, including the ones that say the pipeline is fine and the model needs work. Tell us what your pipeline moves and we will tell you exactly how we would validate it.

• Pick the last ten questions the system answered badly and check, for each, whether the answering document was actually in the index.
• Count documents in the vector store against the source of record. If nobody can produce that number quickly, that is the finding.
• Look for exact and near-duplicate chunks. Most corpora have more than their owners expect.
• Read the ingestion job's retry behaviour, then ask what a partial failure leaves behind.