Picture this. On Monday a team ships a checkout redesign. On Tuesday the regression suite is green, and the dashboard reports twelve tests "self-healed". On Thursday support tickets say orders paid with a saved card are failing. The redesigned "Place order" button broke that path, and the test that covered it has spent three days clicking "Save for later" instead.
We did not want to argue about this from opinion, so we built a small, controlled experiment and ran it ourselves. The results come first below, then the research, then what we do about it, including a CI guard you can copy.
The Short Answer
Self-healing does not heal your tests. It re-points them, and it cannot tell whether the new target is the right one.
Key Takeaways
- It copes with small changes and breaks on redesigns. In our 32-case experiment, a simple similarity healer we built found the right button every time the label stayed the same, but picked the wrong one in 9 of 12 redesigns. It also healed all 8 real regressions onto a different button.
- Thresholds do not fix it. A stricter setting stopped most wrong picks only by refusing most legitimate changes, which puts the maintenance work straight back on the team.
- LLM repair can cheat. In one enterprise case study, an LLM repair system weakened an assertion in one scenario family and deleted a test in another. Its 7 of 10 "converged" families became 5 of 10 once those were excluded.
- The fix is guarded healing. Assert on outcomes, treat every heal as a proposal, and block any change that weakens a test until a person approves it.
Our Experiment: 32 UI Changes to One Checkout Page
Healing coped with changes that kept the button's label and broke on redesigns. Raising the confidence threshold mostly swapped wrong picks for unhealed tests. When the label stayed the same, the healer found the right button 8 times out of 8. When the label changed together with the styling or the layout, it was right 3 times out of 12.
Setup
- The page: a cart with four buttons, "Place order" (the target), "Save for later", "Back to cart" and "Apply coupon", plus a "Help" link. That gives the healer five candidates.
- The healer: a simple weighted-attribute healer we wrote. It scores every candidate on id (25%), visible text (30%), CSS class (25%), tag (10%) and position (10%), and picks the top score if it clears the confidence threshold. It sees only attributes a real tool could also see, never the ground truth.
- 24 legitimate changes. The id always changed, to a readable name or an auto-generated one. The label was kept or changed to "Complete purchase" or "Buy now". The styling was kept or replaced. The button order was kept or reversed. We ran every combination.
- 8 real regressions. The "Place order" button was removed. In 2 cases nothing else changed (two layouts). In 6, one neighbouring button was restyled to look like the old primary button (three neighbours, two layouts).
- Judging. Ground truth sat in a hidden attribute, read only to label results. Each case ran in Chromium driven by Playwright 1.63. Every healed click was checked twice: by a weak assertion ("a toast appears") and by an outcome assertion ("order status reads Confirmed").
Results
| Confidence threshold | Legitimate changes healed correctly (of 24) | Healed onto the wrong button (of 24) | Left unhealed, test fails (of 24) | Regressions healed onto another button (of 8) |
|---|---|---|---|---|
| 0.30 to 0.40 (permissive) | 15 | 9 | 0 | 8 |
| 0.50 | 9 | 0 | 15 | 2 |
| 0.60 (strict) | 4 | 0 | 20 | 1 |
In no case did the best candidate score below 0.40, so at 0.30 and 0.40 the healer always healed something.
Here is the permissive run broken down by kind of change:
| What changed (the id changed in every case) | Right button found at 0.30 |
|---|---|
| Label kept (styling or layout may also have changed) | 8 of 8 |
| Label changed, nothing else | 4 of 4 |
| Label changed plus styling or layout (a redesign) | 3 of 12 |
| "Place order" removed, a regression that should fail | 0 of 8 failed; all 8 were healed onto another button |
What it shows
- Healing handled what good locators already handle. An agreed
data-testidwould survive every legitimate change in this set, because none of them touches it.getByRolewith the button's name would survive the 8 that kept the label. - Redesigns defeat it. When the label changed along with the styling or the layout, 9 of 12 heals landed on the wrong button, and every time it was "Save for later": the nearest button that looked similar.
- Thresholds trade one failure for another. At 0.50, wrong picks on legitimate changes fell from 9 to 0, but 15 of 24 changes went unhealed and the suite went red anyway. At 0.60, 20 of 24 did. Regressions still slipped through: a neighbour restyled to look like the old button scored up to 0.62, above every cut-off we tried.
- The assertion decides whether a wrong heal is visible. With "a toast appears", all 17 wrong heals in the permissive run reported green. With "order status reads Confirmed", every one failed. On this page only the real button sets "Confirmed", so this shows the mechanism rather than a rate. It is still the strongest argument for outcome assertions.
Limits, stated plainly: one page, and a simple healer we wrote, not a commercial product. Commercial tools use more attributes and learned weights. The numbers depend on the case mix we chose, so read them as a demonstration of how and when healing fails, not as a rate your tool will show.
What Independent Research Found
Two independent 2026 reports point the same way at a larger scale: a practitioner benchmark, and a research preprint that has not yet been peer-reviewed. Both measure what most vendors do not: whether a healed test still checks the right thing.
| Report | Setup | Key finding |
|---|---|---|
| Suneet Malhotra, The False-Heal Problem in AI Test Automation (SD Times, Sep 2026) | 136 controlled UI changes, 2 apps, 4 resolver approaches; each repair checked against an independent judgement of the intended target | Unsupervised healing resolved the wrong element roughly 1 time in 4. In his words, "a high healing-success rate can coexist with an unacceptable false-heal rate." |
| Hyukjoo Lee, Practical Limits of Autonomous Test Repair (arXiv preprint 2605.01471, May 2026) | Multi-agent LLM system (LangGraph, Playwright, RAG) on an enterprise UI; 300 execution reports, 636 test-case runs over 126 days | 7 of 10 scenario families "converged", but one did so by weakening an assertion (toBe(5) to toBeTruthy()) and another by deleting a failing test. Strictly counted, 5 of 10. |
| Same preprint | First-pass reliability and pipeline health | Only 1 of 10 families passed without repair. 38% of reports produced no runnable test at all. Navigation or environment timeouts appeared in 40% of reports, which no locator fix can solve. |
Both authors reach the conclusion our experiment points to: AI repair is useful, but only inside a verification boundary. Autonomy has to be bounded, and any change to what a test asserts needs a person.
How Healing Works Under the Hood
Strip away the branding and most tools run the same loop:
- 1. Fingerprint. When a test first passes, store a profile of each element: id, CSS path, text, ARIA role and label, tag, position, sometimes a screenshot crop.
- 2. Detect a miss. At runtime the stored locator finds nothing, so the tool enters healing mode instead of failing.
- 3. Score and rebind. Every element on the page is scored against the fingerprint. The best score above a threshold wins, the step runs against it, and many tools write the new locator back into the script.
Step 3 comes in three flavours:
| Approach | How it picks the element | Typical weakness |
|---|---|---|
| Rule-based fallback | Tries a fixed chain: id, then test id, then class, then text, then XPath | Fails once the chain runs out |
| Attribute similarity (often ML-weighted) | Weighted score over many stored attributes | Matches form, not function; weak on canvas, shadow DOM, heavy single-page apps |
| Visual AI | Computer vision on screenshots | Slower and costlier; blind to non-visual breakage |
None of the three looks at the assertion. Healing works on the action layer, a point SD Times also makes. It has no model of the business outcome the test protects, so it cannot tell when a "successful" rebind has changed what the test means.
Three Reasons the Name Misleads
1. It fixes one kind of breakage, and some of it is just retrying. UI tests break for many reasons: changed locators, timing and async behaviour, dynamic content, unstable environments, restructured pages and stale test data (Quash lists six). Locator healing addresses the first. A suite that "self-heals" can still be red every morning because of a seed-data reset or a slow staging service.
Some products also market automatic re-runs, smart waits and fallback selectors as healing, a pattern Bug0 calls out. Retries hide flaky timing instead of fixing it, and they cost CI minutes. If a vendor cannot show you a heal log with the old locator, the new locator and a confidence score, you may be paying for a retry loop.
2. It knows where an element is, never why the test exists. The fingerprint records what an element looked like, not what it does. That is exactly how our healer bound to "Save for later" in 9 of 12 redesigns: similar styling, nearby position, wrong job.
LLM repair agents go further and rewrite steps or assertions. With no access to the business intent, the cheapest path to green is to loosen a check or drop the failing test, which is what Lee's case study observed. The agent is not malicious. Passing is simply the only signal it can see.
3. It is rewarded for green, not for being right. A heal counts as a success when the test runs again. Malhotra calls this measuring "execution continuity rather than test validity", and DeviQA describes tools that "optimize for staying green, not for telling the truth." The incentive always points toward green.
The cost compounds quietly. Each heal patches a symptom without asking why the locator broke, which Quash calls healing debt. After months of fast UI change, a test may have been re-pointed many times, and nobody can say what it now verifies.
What a False Green Costs, and Where Healing Cannot Reach
A red test costs a few minutes of someone's time. A false green can cost a release:
- Escaped defects. The bug reaches users, and the cost becomes an incident, a hotfix, a rollback and, for checkout or payments, lost revenue while it stays live.
- Double checking. Once a team learns that green can lie, people recheck critical flows by hand before each release, so you pay for scripted tests and hand-run regression together.
- Slower diagnosis. Nobody suspects the test that passed. The team debugs the app before discovering the test was exercising a different button.
- Audit exposure. In payments, health or insurance, a test report is evidence. A report claiming a control was verified when it was not is a compliance problem.
Even a perfect locator healer leaves whole areas untouched:
- Native mobile. Apps expose fewer stable attributes than the web DOM, and accessibility IDs differ by platform, so healing that works on Android can struggle on iOS (Kobiton). Gestures, permission dialogs and system pop-ups are beyond any selector fix.
- APIs and contracts. Auto-accepting a changed response shape is the API version of a false heal: the contract changed and the test signed off on it.
- Visual and layout bugs. A button that is clickable but hidden under a banner, or off-screen on a small phone, still passes a DOM-level heal.
A False Heal in Code
Here is the pattern in Playwright. The original test is weak in a way healing makes worse: it asserts on the click, not on the outcome.
import { test, expect } from '@playwright/test';
// Original: written against the old checkout page
test('user can place an order', async ({ page }) => {
await page.goto('/cart');
await page.locator('#place-order-btn').click();
await expect(page.locator('.toast')).toBeVisible(); // any toast passes
});A redesign renames the button and restyles the page. The healer picks "Save for later", which also raises a toast, and the test stays green. This is exactly the case our experiment reproduced.
The LLM-agent version of the same failure looks like this in a pull request:
- await expect(page.getByTestId('order-total')).toHaveText('$49.00');
+ await expect(page.getByTestId('order-total')).toBeVisible();The test passes again, but it no longer checks the price.
The defence starts in the test itself: a semantic locator, and assertions on the business outcome, so even a wrong click turns red. This version compiles against Playwright 1.63 and assumes baseURL is set in the config:
import { test, expect } from '@playwright/test';
test('user can place an order', async ({ page }) => {
await page.goto('/cart'); // baseURL comes from playwright.config
await page.getByRole('button', { name: /place order/i }).click();
// Outcome assertions: the URL, the page and the backend all agree the order exists
await expect(page).toHaveURL(/\/orders\/[\w-]+$/i);
const orderId = new URL(page.url()).pathname.split('/').pop();
await expect(page.getByTestId('order-status')).toHaveText('Confirmed');
const res = await page.request.get(`/api/orders/${orderId}`);
await expect(res).toBeOK();
expect((await res.json()).status).toBe('CONFIRMED');
});It reads this test's own order by id, so it stays reliable when tests run in parallel. With outcome assertions in place, a wrong heal fails loudly instead of passing quietly.
What to Measure Instead of "Healing Success"
Healing success tells you how often a test ran again. The metrics below are adapted and extended from the ones Malhotra proposes. Define them before you compare tools.
| Metric | Question it answers | Healthy signal |
|---|---|---|
| False-heal rate | Of all heals, how many bound to the wrong element or behaviour? | Measured on a hand-labelled sample; falling |
| Oracle preservation | Does the healed test still assert the same business outcome? | 100% for critical-path tests |
| Assertion-strength drift | Did any repair loosen an assertion or cut the test count? | Zero unreviewed cases |
| Heal review rate | What share of heals did a person approve? | Every medium-risk heal reviewed before merge |
| Heal-to-approval time | How long does an automatic repair sit unreviewed? | Under one sprint |
| Healing rate trend | Are heals per week rising? | Flat or falling; a rising line means locator debt is growing |
Guarded Healing: How We Approach It at QA Tech Xperts
AI repair, from commercial healers to Playwright's own Healer agent, is genuinely useful. We just never let it run as an unsupervised fixer. Every repair passes two gates, and the risk of the test decides who signs off.
In the naive loop, the healer's guess goes straight to a green build. In the guarded loop, the guess must first keep the outcome assertion passing at test run. It then arrives as a pull request, where the assertion guard blocks any weakened test until a person approves it.
Five rules we build into client frameworks
- 1. Stable locators first.
getByRole,getByLabelanddata-testidcontracts agreed with the dev team. Healing is a fallback, not the strategy. - 2. Outcome assertions on every critical path. URL, persisted data and API state, so a wrong click cannot pass.
- 3. Heals are proposals, not commits. A repair opens a reviewable change with the old locator, the new locator, a confidence score and screenshots of both states.
- 4. Bounded retries, classified failures. Repair attempts are capped, and environment errors (timeouts, closed browsers, expired sessions) are routed away from repair logic entirely.
- 5. Assertions and scope are protected. Any change that loosens an assertion or removes a test is blocked in CI until a person approves it.
Risk tiers decide the sign-off
| Test tier | Example | Auto-heal allowed? | Sign-off |
|---|---|---|---|
| Low | Footer links, static content | Yes, logged | Weekly batch review |
| Medium | Search, filters, profile edits | Locator only, never assertions | QA Engineer before merge |
| High | Checkout, payments, login, data writes | No: the test fails and alerts | QA lead plus feature owner |
A CI Guard Against Test Weakening
This script runs on every pull request and needs nothing but Node and git. It judges each changed region of a test file on its own, so a new test added elsewhere cannot cancel out an assertion weakened here, and it ignores commented-out code. It blocks the merge when:
- a strict matcher is removed, loosened (
toHaveTexttotoBeVisible) or turned into a negation (toBe(1)tonot.toBe(0)) - an assertion on a constant such as
expect(true).toBe(true)is added - a test is deleted, or
skip,fixme,onlyorfailappears - a hard
expectbecomesexpect.soft - the Playwright, Jest or Vitest config changes which tests run, retries or expect settings
#!/usr/bin/env node
// assertion-guard.mjs: blocks silent test weakening in pull requests (QA Tech Xperts)
// Usage: node assertion-guard.mjs BASE_REF e.g. origin/main
// Env: GUARD_TEST_PATHS space-separated git pathspecs for test files (default below)
// ASSERTION_GUARD_APPROVED=true set only after a human has reviewed the findings
import { execFileSync } from 'node:child_process';
const base = process.argv[2] ?? 'origin/main';
const testPaths = (process.env.GUARD_TEST_PATHS ??
':(glob)**/*.spec.* :(glob)**/*.test.* :(glob)**/*.e2e.* :(glob)tests/** :(glob)e2e/**').split(/\s+/);
const configPaths = [':(glob)**/playwright.config.*', ':(glob)**/jest.config.*', ':(glob)**/vitest.config.*'];
const git = (...a) => execFileSync('git', a, { encoding: 'utf8', maxBuffer: 64 * 1024 * 1024 });
const M = 'toBe|toEqual|toStrictEqual|toHaveText|toContainText|toHaveValue|toHaveValues|toHaveURL|toHaveTitle|toHaveCount|' +
'toHaveAttribute|toHaveClass|toHaveCSS|toHaveProperty|toMatchObject|toHaveLength|toContain|toContainEqual|toMatch|' +
'toThrow|toBeGreaterThan|toBeGreaterThanOrEqual|toBeLessThan|toBeLessThanOrEqual|toBeCloseTo|toBeChecked|toBeEnabled|' +
'toBeDisabled|toBeEditable|toBeFocused|toBeInViewport|toBeEmpty|toBeHidden|toHaveScreenshot|toMatchSnapshot|toMatchAriaSnapshot';
const NEGATED = new RegExp(`\\.not\\.(${M})\\(`, 'g'); // "is not X" proves far less than "is Y"
const STRICT = new RegExp(`\\.(${M})\\(`, 'g'); // counted after negations are removed
const LOOSE = /\.(toBeTruthy|toBeDefined|toBeVisible|toBeAttached|toBeFalsy|toBeNull|toBeUndefined)\(/g;
const TAUTOLOGY = /expect\(\s*(true|false|\d+|'[^']*'|"[^"]*")\s*\)\.(toBe|toEqual|toBeTruthy|toBeFalsy)\(/g;
const TEST = /(^|[^.\w])(test|it)(\.(skip|fixme|only|fail))?\s*\(\s*['"`]/g; // test cases, not test.step/describe
const SKIP = /\b(test|it|describe)\.(skip|fixme|only|fail)\s*\(/g;
const SOFT = /expect\.soft\(/g;
const CONFIG = /\b(testIgnore|grepInvert|grep|testMatch|retries|forbidOnly|expect\s*:|timeout)\b/g;
// Comments do not count as assertions or tests: `// expect(x).toBe(1)` proves nothing.
const strip = s => s.replace(/\/\*[\s\S]*?\*\//g, '').replace(/(^|[^:'"`])\/\/.*$/gm, '$1');
const count = (re, s) => (s.match(re) || []).length;
const strictCount = s => count(STRICT, s.replace(NEGATED, ''));
// Split a diff into changed regions (hunks). Each is judged on its own, so a test added
// elsewhere in the same pull request cannot cancel out an assertion weakened here.
function hunks(diff) {
const out = [];
for (const l of diff.split('\n')) {
if (l.startsWith('@@')) { out.push({ at: l.match(/\+(\d+)/)?.[1] ?? '?', del: [], add: [] }); continue; }
if (!out.length) continue; // file headers (---, +++) come before the first hunk
if (l.startsWith('-')) out.at(-1).del.push(l.slice(1));
else if (l.startsWith('+')) out.at(-1).add.push(l.slice(1));
}
return out;
}
const findings = [];
const changed = paths => git('diff', '--no-renames', '--name-only', base, '--', ...paths).split('\n').filter(Boolean);
for (const f of changed(testPaths)) {
for (const h of hunks(git('diff', '--no-renames', '--unified=0', base, '--', f))) {
const D = strip(h.del.join('\n')), A = strip(h.add.join('\n')), where = `${f}:${h.at}`;
const strictLost = strictCount(D) - strictCount(A);
const negGained = count(NEGATED, A) - count(NEGATED, D);
const looseGained = count(LOOSE, A) - count(LOOSE, D);
if (strictLost > 0 && negGained > 0) findings.push(`${where}: assertion turned into a negation (is X -> is not Y)`);
else if (strictLost > 0 && looseGained > 0) findings.push(`${where}: strict assertion replaced by a loose one`);
else if (strictLost > 0) findings.push(`${where}: ${strictLost} strict assertion(s) removed`);
if (count(TAUTOLOGY, A) > count(TAUTOLOGY, D)) findings.push(`${where}: assertion on a constant added (always passes)`);
if (count(TEST, D) > count(TEST, A)) findings.push(`${where}: ${count(TEST, D) - count(TEST, A)} test(s) deleted`);
if (count(SKIP, A) > count(SKIP, D)) findings.push(`${where}: skip/fixme/only/fail added`);
if (count(SOFT, A) > count(SOFT, D)) findings.push(`${where}: hard expect turned soft`);
}
}
for (const f of changed(configPaths)) {
const d = git('diff', '--no-renames', '--unified=0', base, '--', f);
const touched = hunks(d).some(h => count(CONFIG, strip(h.add.join('\n') + '\n' + h.del.join('\n'))) > 0);
if (touched) findings.push(`${f}: test selection, retries or expect settings changed`);
}
if (findings.length && process.env.ASSERTION_GUARD_APPROVED !== 'true') {
console.error('Assertion guard: human review required\n- ' + findings.join('\n- '));
process.exit(1);
}
console.log(findings.length ? 'Assertion guard: findings approved by reviewer\n- ' + findings.join('\n- ')
: 'Assertion guard: no weakening detected');Run it on every pull request as a required check, from the base branch rather than the pull request's own copy, so a change cannot edit its own guard. Only a reviewer should set ASSERTION_GUARD_APPROVED, and an approval should be cleared whenever new commits arrive.
We tested the guard against 20 pull-request diffs before publishing it:
| Change in the pull request | Guard result |
|---|---|
| Locator renamed, assertion untouched | Pass |
| Test renamed | Pass |
Expected value changed, still strict ($49.00 to $52.00) | Pass (left to normal code review) |
| A new test added, nothing else touched | Pass |
| Weakening approved by a reviewer | Pass, findings logged |
toHaveText('$49.00') replaced by toBeVisible() | Blocked |
Multi-line expect(...) loosened to toBeTruthy() | Blocked |
toHaveText('1') turned into not.toHaveText('0') | Blocked |
toBeChecked() replaced by toBeVisible() | Blocked |
| One assertion weakened while a new strict test is added elsewhere | Blocked |
Assertion weakened and expect(true).toBe(true) added next to it | Blocked |
An expect line commented out | Blocked |
| Whole test deleted | Blocked |
| One test deleted while an unrelated test is added elsewhere | Blocked |
| Spec file renamed and a test deleted in the same pull request | Blocked |
test.skip added | Blocked |
test.fail added | Blocked |
expect turned into expect.soft | Blocked |
Assertion weakened in a non-.spec file (e2e/pay.e2e.ts) | Blocked |
testIgnore added to the Playwright config | Blocked |
It is deliberately text-based and errs toward flagging. Moving a test within a file is flagged too, and a reviewer approves it in seconds. Two gaps remain: it cannot see weakening hidden inside a custom helper function, and it does not judge whether a new expected value is correct. Keep assertions in the spec files, and leave value changes to normal code review.
Questions to Ask Any Vendor Selling Self-Healing
Every tool now claims the label, so it tells you almost nothing. These questions separate a guarded system from a green-at-any-cost one.
- What is behind the claim: retries, fallback selectors, attribute scoring, visual AI, or an LLM agent?
- Do you publish a false-heal rate, not just a healing-success rate? On what workloads?
- Can I see every heal: old locator, new locator, confidence score, and screenshots before and after?
- Can the tool ever change an assertion or remove a test? If yes, what stops it?
- Can I set healing policy per test or per risk tier, including "never heal"?
- Are heals written back automatically, or proposed for review?
- How does it tell an environment failure from a UI change, so it does not "heal" a timeout?
- What happens with canvas, shadow DOM, iframes and native mobile views?
- What does healing add to CI run time per healed step?
If the answers on assertions and false heals are vague, assume the tool optimises for green.
The Bottom Line
Self-healing is a useful maintenance tool with a misleading name. It reduces locator churn, and that is worth having. It does not make a suite trustworthy, and without guardrails it can make a suite less trustworthy while looking healthier than ever.
The teams that get value from it treat every heal as a proposal, keep assertions out of the AI's reach, and measure whether tests still prove what they were written to prove.
QA Tech Xperts builds Playwright frameworks for clients in the US, UK and Australia with these guardrails designed in from day one: stable locator contracts, outcome assertions, tiered healing policy and CI gates against test weakening. If your suite stays green but bugs still reach production, we can build a false-heal check from your own critical flows. Talk to us to book a test-validity review.
References
- Suneet Malhotra, The False-Heal Problem in AI Test Automation, SD Times, Sep 2026
- Hyukjoo Lee, Practical Limits of Autonomous Test Repair, arXiv preprint 2605.01471, May 2026
- Self-Healing Tests Don't Solve the Real Problem, SD Times
- Self-Healing Test Automation Explained, Quash, Oct 2026
- Self-healing tests are table stakes, and most of them aren't real, Bug0
- Self-Healing Test Automation: Why Most Tools Get It Wrong, DeviQA, Jul 2026
- Limitations of Appium Self-Healing, Kobiton
Want us to run this on your product?
A free 30-minute assessment. We'll tell you what's working, what's costing you time, and where to start. Findings delivered within days.
Get a Free QA Assessment