Testing With AI vs Testing AI: The Complete Guide

Two things get called AI Testing and they are not the same job. One is using AI to help test a conventional product. The other is Testing a product whose behaviour is produced by a model.

A QA Engineer asking an assistant to draft a Playwright spec is doing the first. A QA Engineer checking whether an agent retrieved the right document, called the right tool and described what it actually did is doing the second. Conflating them is why teams buy the wrong tool.

Two Disciplines That Share a Name

The cleanest way to hold the distinction is to ask where the AI sits. If it sits beside the tester, helping produce artefacts, that is Testing with AI. If it sits inside the product, producing behaviour a user depends on, that is Testing AI.

Testing With AI: The Assistant in the Loop

Here the product is ordinary. E-commerce, banking, SaaS, APIs, OTT. Nothing about the system under test has changed; what changed is how quickly the team can produce the things it needs to test with.

The useful applications are unglamorous and they compound. The constraint is that none of the output is evidence until somebody qualified has looked at it.

• Requirement analysis and test-case ideation, including the negative and boundary cases a tired person skips.
• Drafting Automation in Playwright, Selenium, Cypress or Appium, and API suites in Postman or Karate.
• Test-data generation for boundary values, international formats, duplicates and unusual combinations.
• Summarising a wall of failures into a small number of probable causes.
• Maintaining tests after a UI change, which is where most Automation budget quietly goes.

What AI-Generated Tests Get Wrong

A generated test is an engineering artefact, not proof of correctness. It arrives looking finished, which is exactly what makes it harder to review than code someone struggled to write.

Two cautions worth stating plainly. Generated coverage is not the same as considered coverage, and production data should not be pasted into a third-party assistant without the authorization and handling controls that would apply anywhere else.

• Assertions that check a page rendered rather than that the right thing happened.
• Expected results inferred from the implementation rather than from the requirement.
• Fragile selectors that pass today and break on the next redesign.
• Redundant cases that inflate the count while covering one risk five times.
• Missing negative paths, because the happy path is what the prompt described.

Testing AI: Where HTTP 200 Stops Meaning Anything

Now the model is inside the product, and the traditional signal disappears. A request can succeed, validate against its schema and return a well-formed answer that is simply wrong. Nothing crashes, because from the system's point of view nothing went wrong.

User
  -> Web / Mobile UI
     -> Application API
        -> LLM
           -> RAG / knowledge base
              -> External tools
                 -> Business logic
                    -> Response      HTTP 200

• Incorrect information, delivered fluently and with confidence.
• Claims unsupported by anything the system actually retrieved.
• An answer to a question that was not the one asked.
• Content that breaches a safety or policy requirement.
• Information the requesting user was never authorised to see.
• A tool called with the wrong arguments, or the wrong tool entirely.

What You Actually Evaluate

Hallucination gets the attention, but it is one item on a longer list. What belongs on your list depends on what the system does and what it is allowed to touch.

The NIST AI Risk Management Framework organises this around characteristics including validity and reliability, safety, security and resilience, privacy and fairness, applied according to context rather than uniformly.

• Functional correctness: did it perform the task it was asked to perform.
• Accuracy and relevance: is the information right, and does it answer the actual question.
• Groundedness: is every claim supported by what the system retrieved.
• Consistency and robustness: equivalent inputs behave equivalently, and odd inputs do not break it.
• Safety and security: it stays inside policy, and cannot be manipulated into leaving it.
• Privacy: confidential information cannot be surfaced to someone without rights to it.
• Agent behaviour: planning, tool selection, recovery and an honest account of what happened.

Define Good Before You Measure It

Most AI quality programmes stall because nobody wrote down what a correct answer must do. Does the answer look good is not a test. The requirements below are, and they are the input to every dataset and harness that follows.

The answer MUST be drawn from approved sources.
The answer MUST NOT invent citations or policy numbers.
The answer MUST NOT expose information the caller cannot access.
The answer MUST address the question that was asked.
The answer MUST follow the required response format.
The answer MUST refuse requests outside the permitted scope.

The Evaluation Dataset Is the Deliverable

An evaluation dataset is real inputs paired with what a correct response looks like, reviewed by somebody who knows the domain. It is the asset the whole programme rests on, and it should be versioned alongside the harness and grow whenever production surprises you.

RAG Adds a Layer That Fails on Its Own

Retrieval-augmented generation puts a search problem in front of a language problem, and they fail independently. Evaluating them together produces a number that tells you something is wrong without telling you which half.

A strong model cannot compensate for consistently poor retrieval. If the right document never arrived, a faithful answer to the wrong document is the best outcome available.

Question
  -> Embedding
     -> Vector / hybrid search      <-- evaluate retrieval here
        -> Retrieved documents
           -> LLM                   <-- evaluate generation here
              -> Answer

• Retrieval: context precision, context recall, relevance of what came back, sensitivity to noise.
• Generation: faithfulness to the retrieved context, response relevance, answer accuracy.
• Measure them separately, then look at the pair. A high generation score over weak retrieval is a false comfort.
• Our field notes on the data side of this are in data quality for RAG systems.

Testing AI Agents

An agent understands a request, plans, selects a tool, builds its arguments, reads the result, decides what to do next, and eventually answers. Every one of those is a place to be wrong, and the final answer can look fine while a step in the middle did not.

Which is the failure worth internalising: the task can run to completion, produce a well-formed response, and report success, while one step silently did the wrong thing.

• Goal achievement: did the task actually get done, as opposed to described as done.
• Planning: was the sequence of actions a reasonable way to reach the goal.
• Tool selection and arguments: the right tool, called correctly.
• Authorization: was the action permitted for this caller, checked outside the model.
• State and context: did it keep track across steps.
• Failure handling and loop control: what it does when a tool fails, and whether it knows to stop.
• Final response honesty: does the answer accurately describe what happened.

Prompt Injection and Sensitive Information

The OWASP Top 10 for LLM Applications lists prompt injection as LLM01 and sensitive information disclosure as LLM02. Both are QA concerns as much as security ones, because both are reachable from ordinary application inputs.

Retrieval does not remove injection risk, it widens it. A poisoned document inside a knowledge base is untrusted content that the system was designed to read and trust.

• Direct injection in user input, and indirect injection through retrieved or uploaded documents.
• Instruction conflicts, where user text argues with the system prompt.
• Attempts to extract system instructions or tool definitions.
• Tool-related injection, where content steers which tool gets called.
• Cross-user access: can caller A reach caller B's records through the assistant.
• Leakage into places nobody reviews, such as traces, logs and error messages.

Authorization Belongs in the Application

This is the architectural point the rest of the security work depends on, and it is worth stating on its own. A model instructed not to reveal something is a model that has been asked nicely.

Access control has to be enforced where it cannot be argued with: in the application, on the retrieval query, and on every tool the agent can reach. The evaluation dataset then proves it, with cases that try to get past it.

WRONG   system prompt: never show another customer's data
        -> one convincing message away from failing

RIGHT   retrieval filtered by caller identity before search
        tool arguments validated against caller permissions
        model physically cannot see what it must not reveal

Red Teaming, and Why It Is Not the Whole Programme

Red teaming is structured adversarial Testing aimed at finding the weaknesses systematic evaluation did not think to look for. NIST describes it as a structured effort to identify flaws and vulnerabilities in AI systems.

It complements evaluation rather than replacing it. Red teaming finds the case nobody wrote down; the dataset makes sure that case is never lost again once found.

• Jailbreak and injection attempts, including multi-turn attempts that build slowly.
• Data leakage and privacy attacks against the retrieval layer.
• Excessive agency: can it be talked into an action beyond its remit.
• Tooling worth knowing: Promptfoo for evaluation and red teaming, Garak as an open-source vulnerability scanner, Microsoft PyRIT for generative-AI risk identification.
• Guidance worth reading: the OWASP AI Testing Guide, and the NIST AI RMF with its ARIA evaluation programme.

LLM-as-a-Judge Needs Company

Using a model to grade another model handles the criteria a regex cannot: was this grounded, was it relevant, did it invent a condition. That is genuinely useful and it is not sufficient on its own.

The judge can be wrong in the same directions as the system it grades. For anything high-risk, combine it with checks that cannot drift.

• Deterministic assertions for anything with a correct answer you can name.
• Reference-based comparison against the expected response in the dataset.
• More than one judge, or more than one prompt, where the stakes justify it.
• Human and domain-expert review on a sample, permanently rather than at launch only.

Deterministic Testing Did Not Go Away

A large part of an AI product is not AI. Authentication, persistence, permissions, billing and the API surface are all ordinary software with ordinary correct answers, and an evaluation harness is the wrong instrument for them.

assert response.status_code == 200
assert transaction_id is not None
assert returned_user_id == authenticated_user_id
assert retrieved_docs.owner == caller.tenant

• Use assertions wherever the requirement can be stated as one. They are cheaper, faster and never ambiguous.
• Add evaluation only where an assertion cannot express the requirement.
• A mature programme runs both, and knows which question each one answers.

Offline and Online, Because Release Is Not the End

Offline evaluation runs the dataset before release and gates on the result. Online evaluation watches the real thing, because production asks questions no dataset anticipated and that is where the next dataset entries come from.

• Before release: regression evaluation, safety cases, RAG and agent evaluation, prompt comparison.
• After release: quality trend, failure clustering, user feedback, tool failure rates, latency and cost.
• Feed production surprises back into the dataset. That loop is the programme; everything else is setup.

AI Regression Testing Asks a Different Question

Conventional regression asks whether the release broke what worked. AI regression asks whether a model, prompt, retrieval or tool change degraded behaviour that was previously acceptable, which is a question about a distribution rather than a build.

A drop in the headline number is the start of the investigation, not its conclusion.

Version A    100 cases    92 acceptable
Version B    100 cases    84 acceptable

Which 8 moved?          Which risk category?
Retrieval or generation? Did safety improve while accuracy fell?
Is the threshold right for what this system is allowed to do?

• Track per-category rates, not one aggregate. An aggregate hides a safety regression behind an accuracy gain.
• Set thresholds from the system's risk, not from a round number.
• Keep every previously failing case forever. That is what makes the suite compound.

The Toolbox

Tools move quickly and this table will age. Check current documentation before committing a production programme to any of them, and treat the categories as more durable than the names in them.

The Skill Set This Creates

None of the conventional discipline becomes obsolete. Functional, API, database, Automation, performance, security and accessibility Testing are all still the job, and CI/CD is still where they run.

What is added is a way of thinking about systems that do not behave deterministically, and it is a smaller leap than it sounds for anyone who already reasons about risk rather than about test counts.

• Designing evaluation datasets, which is closer to test design than to data science.
• Reading a distribution rather than a pass or fail, and knowing what a moved rate means.
• Prompt and retrieval evaluation as distinct skills with distinct failure modes.
• AI security: injection, leakage, excessive agency and the authorization boundary.
• Observability, because the evidence you need lives in traces rather than in a report.

The goal is not to trust an AI system because it produced an impressive answer. It is to hold measurable evidence that it behaves inside its requirements, its operating conditions and its risk boundaries, and to keep holding it after the next model change.

We build this with product teams as a practice rather than an audit: evaluation datasets, RAG and agent harnesses, injection and authorization Testing, and the release gates that use them. See our AI Quality Engineering practice, or tell us what you are building and we will tell you where we would look first.