On this page
Self healing test automation is sold as less maintenance, and it delivers: a button gets a new ID, the test finds it anyway, and nobody spends Tuesday fixing selectors. The catch is how: it makes the test agree with whatever the application now does. That is right for a renamed button, and the worst thing a test can do for a checkout that now charges twice.
This post is about telling those two cases apart. Healing is one layer of AI test automation, and worth having. It just answers a different question from the one a failing test asks.
What is self-healing test automation?
Self-healing test automation is a runner that repairs a failing test instead of reporting it. It does three things: re-finds an element whose selector broke, stretches waits when a step times out, and, in some setups, loosens an assertion whose expected value no longer matches. What changed is recorded, if at all, in a log.
That last sentence is the whole argument. Self healing tests keep the signal green. Whether anyone reads why is a process question.
The three mechanisms
Self healing test automation tools all do some mix of three repairs, and the three carry very different risks.
| Mechanism | What it repairs | What it can hide |
|---|---|---|
| Selector repair | An element whose ID, class, text or position changed; the runner picks the best-scoring nearby candidate | The wrong element: the first line item instead of the total, or an old button that is no longer wired |
| Wait and timing adjustment | A step that timed out; the runner waits longer or retries | A real slowdown, or a request that now takes two round trips |
| Assertion softening | An expected value that no longer matches; the check widens to a pattern, a range or the new value | A changed number: the test now asserts the bug |
Selector repair is the most common need. In a study of 1,065 breakages across eight open-source web apps, Hammoudi, Rothermel and Tonella found that "locators caused over 73% of the test breakages we observed" (ICST 2016, summarised by The Morning Paper). That is the biggest pile, and why automated test healing caught on.
The masking problem in self healing test automation
A selector breaks for one of two reasons: the markup changed and the behaviour did not, or the behaviour changed and the markup changed with it. Selector repair cannot tell which. It finds the closest element and carries on.
One checkout test, two releases.
Release A renames the button. #pay-btn labelled "Pay" becomes #checkout-submit labelled "Place order". Same handler, same request. Healing re-finds the button, the test passes, and that pass is true.
Release B changes the total. A refactor splits shipping into its own charge, so the card is charged twice and the one .order-total of $42.00 becomes two lines, $42.00 and $6.99. The old selector is gone, so the healer picks the closest match, the first line, and reads $42.00. The assertion passes.
Both runs are green, and both reports say the same thing: one healed step, logged at info level. Release B ships a double charge with a passing test as its alibi.
Had the setup softened the assertion to "a dollar amount is shown" instead, the outcome is the same: a loud failure became a quiet one. Retries do the same thing to flaky tests, and teach the team the same habit.
Three cases where healing is the right call
None of this makes self healing automation a bad idea. In three situations it is the correct tool, and refusing it just burns time.
Cosmetic refactors. Class names change in a CSS rewrite, generated IDs change between builds, wrappers appear around buttons. Behaviour is identical, and a test that fails on a renamed class is testing the stylesheet. Heal these and move on.
Framework or design-system migrations. Moving a few hundred screens to a new component library touches every selector at once. Without repair, the suite is red for weeks and nobody trusts it by the end. Healing carries it through while the markup churns.
Third-party DOM you do not control. Payment iframes, chat widgets and identity-provider login pages change markup on someone else's schedule. You cannot pin it, and you mostly care that the step completes. Repair the selector, and check your own side of the exchange: the token came back, the session exists.
In all three, the markup moved and the behaviour did not. That is the condition healing assumes, and when it holds, healing is pure saving.
The alternative: re-derive the verdict
The fix for masking is to stop letting the selector decide the verdict. If the assertion reads what the application did (which requests fired, what the store holds), finding the button and judging the outcome become separate jobs. The verdict is computed fresh from recorded evidence, which is the core idea of AI verification: judge the outcome, not the screen.
Deterministic replay works this way. You record the path once, with the expected consequence of each step. Every run replays it, collects network and state again, and re-derives pass or fail. Nothing is repaired: if the evidence differs, the run fails and names the difference.
Before, a check on what the page shows, which a healer can satisfy by picking a different element:
await click("#pay-btn"); // healed to #checkout-submit
await expectText(".order-total", "$42.00"); // healed to the first line item
After, a check on what the app did, which no element choice can satisfy. The helper names are illustrative; the shape is what matters:
await click(byRole("button", /pay|place order/i));
expect(requests("POST /api/charge")).toHaveLength(1); // release B: 2
expect(state("order.total")).toBe(4200); // release B: 4899
expect(state("order.status")).toBe("paid");
Release A passes both. Release B passes the first and fails the second on two lines, each saying what changed. The second test does not care what the button is called, only that one click made one charge for the right amount.
The failure modes this catches, from double submits to swallowed errors, are listed in seven failure modes of AI-generated code. This post does not repeat them.
Cost and stability, side by side
| Self-healing | Deterministic replay | |
|---|---|---|
| Maintenance effort | Low for markup churn; healed steps should still be reviewed | Low for markup churn if the step targets role and text; a changed outcome needs a re-recording |
| Flake rate | Not measured here; each run re-searches the page, so a match can vary between runs | 0% in our benchmark: eight replays of one flow, one verdict |
| What a failure tells you | Usually that no candidate scored high enough | Which request, state value or console line differed from the recording |
| Cost per re-run | Depends on the tool; model-based matching spends tokens on each healed step | About 480 tokens to re-verify a four-flow suite, with no model in the loop |
The two figures come from the benchmark below. We have not measured a healing tool's flake rate, so that column states only the mechanism. For what other kinds of tool read to decide pass or fail, see our comparison of regression testing tools.
A decision rule
Ask of each failure: did the behaviour change, or only the markup?
| Situation | Use healing | Re-derive the verdict |
|---|---|---|
| Class names and IDs changed in a refactor | Yes | Not needed |
| Migrating to a new component library | Yes, through the migration | Keep the outcome checks running alongside |
| A third-party widget changed its markup | Yes, for the widget's own steps | Yes, for your side of the exchange |
| A price, total or quantity changed | No | Yes |
| A request count or endpoint changed | No | Yes |
| A test fails and nobody knows why | No: healing would hide the answer | Yes: the diff is the answer |
The short version: heal the path, never the verdict.
Reticle is built on the right-hand column: it records flows inside the running app, with network, state and console for each step, and replays them with no model in the loop. In its published benchmark, re-verifying a four-flow suite costs about 480 tokens, and eight replays of one flow gave one verdict (0% flake). It does not replace healing for markup churn, and a person still decides the right total.
Fun fact
"Self-healing" did not start in testing. It is one of the four self-management properties of autonomic computing, an initiative IBM launched in 2001 and named after the autonomic nervous system, the part of your body that keeps your heart beating without you noticing (Wikipedia).
The property is summarised there as "automatic discovery, and correction of faults". Discovery comes first. Test healing kept the correction and quietly dropped the part where someone finds out.
Conclusion
Self-healing tests solve a real problem: most breakages are selectors, and most selector changes are cosmetic. But a healed test agrees with the application, which is only safe when the behaviour did not change. Heal the path through markup churn, migrations and third-party widgets. For money, counts and state, re-derive the verdict from what the app did.
FAQ
What is self-healing test automation?
It is a test runner that repairs failing tests instead of reporting them. It re-finds elements whose selectors broke, adjusts waits when steps time out, and sometimes loosens assertions whose expected values changed. The goal is lower maintenance. The cost is that a repaired test agrees with the app, whether or not the app is right.
Is self-healing test automation reliable?
It is reliable at keeping tests running through markup changes, which cause most breakages. It is not reliable at telling you whether the behaviour is still correct, because a repair can match the wrong element or accept a changed value. Review healed steps, and pair healing with assertions on network and application state.
Does self-healing fix flaky tests?
It hides some of them. Timing repair absorbs slow runs, so a timing flake stops failing, but the cause is still there and the test still reads the wrong signal. A flaky test is fixed by asserting on an event or a state the code produces. Healing reduces the red without removing the coin flip.
When should you not use self-healing?
Do not let it repair a step whose outcome matters: a total, a charge, a request count, a permission check or anything stored. Do not use it when a test fails for an unknown reason, because the repair erases the evidence. Keep it for cosmetic refactors, design-system migrations and third-party markup you do not control.
