The Challenge
AI features fail differently: hallucinations, prompt regressions, and RAG retrieval errors that no traditional test suite flags. A wrong answer returns HTTP 200. We needed a repeatable way to measure whether an LLM change made things better or worse before we could credibly offer that to anyone else.
What We Built
QEAP, our AI Quality Engineering harness: LLM output evaluation, prompt regression Testing, RAG retrieval accuracy checks, grounding verification and hallucination detection, packaged so it can be stood up inside a client's pipeline rather than run as a service by us.
Why we built it before selling it
A conventional test suite can be perfectly healthy and completely blind to how an AI feature fails. A wrong answer returns HTTP 200. A prompt edit that fixes one complaint quietly degrades ten other behaviours, and nobody finds out until a user writes in. Most teams shipping LLM features decide releases by trying a few prompts by hand and forming an impression.
That is the default state of the field, not negligence — the discipline is new and the failure modes are invisible to error monitoring. But we were not willing to offer AI Quality Engineering as a service while our own answer to "how would you measure that" was a slide. So we built the harness first, on our own time, and it is what we now stand up inside client pipelines.
Where every AI evaluation has to start: the dataset
The unglamorous work determines whether everything downstream is real. An evaluation suite needs a golden dataset built from actual usage rather than imagined queries — the phrasing people genuinely type, misspellings included. For each question you record the document chunks a correct answer requires, and what a correct answer must and must not claim.
On a client engagement that dataset is built from their support transcripts and search logs, and validated by a subject-matter expert on their side. Without that validation step, an evaluation suite measures nothing but its own assumptions. This is the part we will not let a client skip, and the reason our AI engagements start with an audit rather than with tooling.
The four layers it makes measurable
- Retrieval accuracy: recall@k against the golden dataset, re-run on every reindex and embedding change — the metric most teams never have.
- Prompt regression: a fixed battery of inputs with expected-property assertions, running on every prompt edit exactly like unit tests run on every commit.
- Grounding: every factual claim traced back to a retrieved chunk, so an unsupported claim is flagged as a hallucination even when it happens to be correct.
- Format contracts: schema validation on structured outputs in both test and production, with visibility into parse-repair rates.
Turning evaluation into a release gate
A harness that runs when someone remembers it is a research project, not a quality practice. What makes it stick is wiring the suite into CI so a prompt, model or index change cannot merge without its numbers being checked against the baseline, and putting those numbers somewhere the whole team sees them.
The cultural shift follows the tooling. Once a better answer is something you can show, "it felt better in the demo" stops being an argument anyone makes.
What a client keeps
When we stand this up for a product team, the harness, the golden dataset, the CI configuration and the runbook for extending all three are theirs, in their repositories, under their licence. The dataset in particular compounds: every production failure they diagnose becomes a new case, so evaluation coverage grows from reality rather than from guesswork.
We are not trying to keep the capability. A harness only we can operate is a dependency, and a dependency is not a quality practice.
Why the dataset is the whole problem
The transferable lesson from building this is that the harness is the easy part. Scoring outputs is engineering. Agreeing what a correct output is turns out to be an organisational problem, and it is where these projects genuinely succeed or quietly fail.
That is why we start there on every AI Quality Engineering engagement, including the ones where the client arrives asking for tooling. A harness built on an unexamined definition of correct measures its own assumptions very precisely — which is worse than measuring nothing, because it produces confidence.
What we would do differently
We would instrument production sampling earlier. A dataset assembled from historical transcripts and search logs is a good start and still a backward-looking view; live sampling of real inputs and outputs surfaces drift weeks sooner and makes the dataset grow continuously rather than in scheduled bursts. We would also put a cost threshold in the gate from day one alongside the quality ones, because token spend behaves exactly like a performance regression: it degrades gradually, nobody owns it, and it is noticed on an invoice. The method is written up in full in How to Test a RAG Pipeline.
The Results
- · LLM changes are measured rather than eyeballed — every prompt change runs the evaluation suite
- · RAG retrieval accuracy is tracked against a golden dataset, re-run on every reindex
- · Grounding and format checks run before an AI change can merge
- · The harness is what we stand up on client engagements, not a demo we keep to ourselves