AI test automation that does not rot

Read the top guides to AI test automation and they share one assumption: the expensive part of testing was writing the tests. So the pitch is speed: describe a journey, get a test in seconds, let the model repair it when the UI moves.

All of that works. It is also why AI-written suites go stale faster than hand-written ones, because the authoring was never what made a test worth having.

A test is worth having when its green means something. That depends on one decision: was what happened correct? Typing is now cheap. That decision is not.

What is AI test automation?

AI test automation is the use of machine learning models across four layers of a test suite's life: generating tests, keeping their selectors working as the UI changes, judging whether each run's outcome was correct, and maintaining the suite as the product moves. Most products automate the first two. The third decides whether a green result means anything.

Defined as layers, the category stops being one thing you buy. You can ask any setup which of the four it does.

The four layers

Most AI test automation tools sell the first two layers and a slice of the fourth. The third is the one the buyer needed.

Layer 1, generation: genuinely good

Give a model a running app and "a returning customer buys two items with a saved card", and it produces a reasonable test: navigation, form fills, a click on Pay, an assertion that a confirmation appears. Give it a currency field and it lists the cases a tired person skips: zero, negative, a switch after the cart fills.

This is why every demo of AI automation testing looks impressive. Generation is the visible part, and it is fast.

What a model cannot know is what "correct" means for your product. It sees a confirmation, so it asserts the confirmation. It does not know the charge should fire once, or that the order should carry the discounted total.

Those facts live in a ticket, in a person's head or nowhere. A generated assertion is a guess about intent, written in the confident syntax of a spec. Feeding the generator recorded flows instead of a prompt narrows the guess; AI test case generation shows how.

Layer 2, selector healing: the masking problem

Healing is solved in the sense that the test keeps running: when #pay-btn becomes #checkout-submit, the tool finds "the Pay button" anyway. The catch is that a test which heals to the wrong element still passes, and a test that cannot fail on a changed UI has stopped testing the UI. Healing turns a loud failure into a silent one, so it needs a judgement layer underneath it more than any other layer does.

Self-healing tests works through one checkout and two releases, and the three cases where healing is the right call.

Layer 3, judgement: the layer nobody automates

Testing has a name for this layer. The thing that decides whether an output is correct is called the test oracle, a term William Howden introduced in 1978 (Howden, IEEE TSE). Deciding what the correct output should be is known as the oracle problem, and it has a whole survey to itself, covering the research from 1978 to 2012 (Barr et al., IEEE TSE).

AI test automation did not solve the oracle problem. It mostly moved it. The assertion used to be written by a person who knew the intent. Now it is written by the same model that wrote the test, and sometimes by the same model that wrote the change.

Why the author cannot be the judge

An agent that just implemented a feature is now asked whether its own choices were right. It looks where its reasoning points, and its blind spots are exactly where the bug is. Code review exists for the same reason: the author is the worst-placed person to spot what they got wrong, and a model is no exception.

A model asked to judge a run also judges from what it can see, which is usually the screen and the text of the page. Most of what makes a run correct never reaches the screen.

A confident green over mock data

Picture the failure, because it is a common shape. A team ships a checkout change written by a coding agent. The agent also generates the test: add two items, pay with a saved card, assert "Order confirmed". It runs green, the agent reports done, the pull request merges.

In the dev environment, a request-mocking handler from an earlier experiment was still enabled. The Pay click never reached the payments service: the handler returned { "status": "ok" }, the UI rendered the confirmation, and the order stayed pending.

Every assertion in the test was true. The product was broken.

Better generation does not fix this, and healing makes no difference. The test asked the wrong question.

What a verdict has to carry

A verdict worth trusting answers "did the right things happen?" from evidence, not from the screen alone. For a web app that means five things:

1. DOM outcome. What the user saw: the confirmation, the error, the disabled button.
2. Network result. Which requests fired, to which host, how many times, with what status. One charge, not two. To the real service, not a mock.
3. Application state. What the store or database holds afterwards: the order is paid, the cart is empty.
4. Console. No uncaught errors, no failed hydration hiding behind a working page.
5. Source location. When it fails, the file:line that caused it, so the fix starts in the right place.

A person decides once what correct means for a journey. The check then applies that definition the same way on every run. That split keeps the judgement human and the execution mechanical, which is the part of AI verification that automation can safely own.

Reticle is a reference implementation of this layer. It runs inside the app in development, so a verdict reads network, state, console and the DOM together and names the source line. One call, from the project README:

// The agent clicked "Pay". Did the right things actually happen?
reticle_assert({
  predicate: { kind: "allOf", predicates: [
    { kind: "net",     method: "POST", urlContains: "/api/order", status: 200 },
    { kind: "element", query: { role: "dialog", name: "Order confirmed" }, state: "visible" },
    { kind: "signal",  name: "order:saved" },          // the charge actually committed
    { kind: "console", level: "error", absent: true }  // …and nothing errored
  ]}
})
// → { verified: "no",
//     because: "the declared consequence did not hold",
//     pass: false,
//     failureReason: "POST /api/order returned 500, expected 200",
//     source: "src/checkout/PayButton.tsx:42" }

In its benchmark of 88 seeded bugs it caught 85 with no false alarms, against 59 for a Playwright script on the same app (false-green scorecard). The gap sits in bugs the screen hides: silent 500s, a store that disagrees with the UI, a double submit.

Layer 4, maintenance: what rots, and in what order

Suites rot in a predictable sequence.

Selectors go first. This is the part healing addresses, and the part everyone measures.

Fixtures and data go next. A seeded user loses a permission, a test card expires, a feature flag flips. The test still runs and now exercises a path nobody meant.

Assertions drift from intent third. The product changes on purpose, and the old assertion either fails loudly, which is fine, or keeps passing because it checked something incidental. The second kind is the dangerous kind.

Meaning goes last. Two years in, nobody can say what a green run covers, so the suite gets rerun until it passes, and flaky tests get a retry and no fix.

The maintenance tax is the time spent reviewing healed selectors, regenerating drifted tests and triaging reds that turn out to be noise. Nobody quotes it, because it arrives as engineering hours rather than an invoice. Generated suites pay more of it: they are larger, and nobody wrote their assertions by hand.

The cost model of a re-run

A suite's real job is to run the same checks again on every commit. That makes the re-run cost the number that compounds.

The replay figure is about 251 times cheaper than a model re-drive, from the 2026-10-06 run of the benchmark scorecard. The same scorecard records one verdict across eight replays of one flow: 0% flake. Its own caveat matters, and it is in the table: a compiled script costs no tokens at all. The claim is cheaper judgement than a model, not cheaper than code.

Cost also changes behaviour. Automated AI testing that costs a fraction of a cent runs on every commit. A check that costs dollars runs when someone remembers.

Build versus buy

You can assemble much of what AI based test automation tools sell from open-source parts:

• Driving: a browser automation library, or an agent on top of one, explores and records.
• Network evidence: request interception or HAR capture records what fired and what came back.
• State: most stores expose a dev hook, so a test can snapshot state after each step.
• Source location: source maps and a build plugin can map a DOM node back to a component file.

The hard part is joining them. Tying each response to the click that caused it, reading framework state live, and replaying a path with the clock controlled is a product, not a weekend. An AI test automation framework you build yourself will do the first three layers well and leave you to maintain the join.

Three questions decide it for a given team:

1. Do you own the app's build? Evidence from inside the app needs a dev dependency. If you can add one, the judgement layer is available to you. If you test apps you cannot modify, you are mostly limited to the screen.
2. How often do you re-run? A suite run weekly can afford a model re-drive. One run on every pull request across many agents cannot.
3. Who defines "correct"? Somebody has to write down, per journey, the consequences that must hold. If nobody will, no tool fixes that, bought or built.

If those answers point to buying, you can start free on the hosted platform, or install the open-source SDK and keep everything on your machine.

Where this approach loses

A judgement layer built on app evidence is the wrong tool in specific places.

Migrating from Selenium or Playwright

You do not need to throw out a working suite. If you are weighing a different runner instead, Playwright alternatives compares four of them on what each one checks.

Keep: unit tests, visual and cross-browser tests, and any end-to-end test whose assertions someone can still explain. Keep your seed data and CI wiring too. To see which regression testing tools cover which failures, compare what each one reads to decide pass or fail; the wider field is in our list of best AI testing tools.

Record: the journeys where a silent failure costs money or trust. Checkout, signup, permissions, anything that writes data. For each one, write down the consequences that must hold, then record the flow once and replay it on every change. A testing harness that replays it decides what it can catch, so put the declared consequences in the recording, not in a comment.

Retire: tests that only assert a text is visible after a fixed sleep, tests whose healed selectors nobody reviewed, and generated tests nobody can explain. A test that cannot fail for the right reason is maintenance you pay for nothing.

Run both for a few weeks. When the recorded flows catch what the old tests catch, delete the duplicates.

Fun fact

The idea of checking a program by building a second one is older than most test frameworks. In 1982 Elaine Weyuker described programs for which no oracle exists as "non-testable" and proposed a workaround: a pseudo-oracle, a separately written program that takes the same input, so the two outputs can be compared (Weyuker, The Computer Journal; test oracle).

The key word was "separately". Asking the model that wrote the code to grade it is a pseudo-oracle with the independence removed.

Conclusion

AI test automation solved typing. Generation and selector healing work, and maintenance is a pricing question. Judgement, deciding whether what happened was correct, is still the open layer, and it is the one that decides whether the rest is worth paying for.

Put a person's definition of correct into declared consequences, check them against evidence from inside the app, and replay them without a model. Use screenshots and cross-browser grids where state cannot see.

FAQ

What is AI test automation?

AI test automation is the use of machine learning models across four layers of a test suite: generating tests, healing selectors when the UI changes, judging whether each run's outcome was correct, and maintaining the suite. Most tools automate generation and healing. Judgement is the layer that decides whether a green result means anything.

Does AI test automation replace testers?

No. It replaces typing, not judgement. Models write and repair tests quickly, but someone still has to define what a correct outcome is for each journey and review what the model produced. The tester's job moves from writing scripts to deciding what must be true and checking that the suite proves it.

Why do AI-generated tests break?

AI-generated tests break in two ways. Loudly, when the UI or data changes. Quietly, when healing re-targets the wrong element or an assertion checks something incidental, so the test passes while the product is broken. The quiet way is worse, because nothing tells you, and it grows with the size of a generated suite.

How much does it cost to re-run a test suite?

It depends on what decides pass or fail. In one published benchmark, a model re-driving a four-flow suite used about 120,000 tokens per run, while replaying the recorded flows with no model used about 480. A compiled script costs no tokens, but checks only the assertions someone wrote.