Automated testing: a field guide for agent-written code

Most guides to automated testing describe the same four steps: write a script, run it, compare the result, report. They are right, and they skip the part that changed. When a coding agent writes the code and the tests, all four steps still run. What moved is who decides what the tests should check.

This guide maps that boundary. Where a topic already has its own post, it links there instead of repeating it: the testing harness that runs the checks, the smoke testing gate on every commit, and the regression testing tools compared by what they can see. To build one small verification harness by hand first, there is a tutorial for that too.

What is automated testing?

Automated testing is software that runs checks a person specified in advance and reports whether each one held. It can repeat a decision forever, at any scale, for pennies. It cannot make a decision nobody wrote down. Everything useful about a test suite, and every way it misleads you, follows from that boundary.

A test is a decision made once and replayed. expect(total).toBe(10800) settled what the right total is on the day someone typed 10800. Every run after that checks the code against that person's understanding of the requirement. If the understanding was wrong, the suite will defend the mistake indefinitely, and very efficiently.

What automation actually automates

Most definitions treat automated testing as one thing. It is four jobs, and only one of them is fully done by machines.

Execution has been solved for decades. Test runners, CI servers and parallel workers do it better than any person. The other three are partly automated, and the part left over is always the same: deciding what correct means.

Maintenance shows it most clearly. Jest's docs describe jest --updateSnapshot, or -u, which "will re-generate snapshot artifacts for all failing snapshot tests" (Jest). It is a useful command. It also automates maintenance by deleting the judgement: every failing snapshot becomes the new truth, whether the change was intended or a bug.

The AI test automation guide splits the same work by what AI tools add to it. This table asks the older question: which parts were ever automatable at all.

Why the test pyramid is the wrong model for agent-written code

The test pyramid says: many fast unit tests, fewer integration tests, very few end-to-end tests. It rests on a stated assumption, that broad tests are "expensive, slow, and brittle", and Martin Fowler names the exception himself: "If my high level tests are fast, reliable, and cheap to modify - then lower-level tests aren't needed."

The industry has bent the shape before, for cost reasons. Guillermo Rauch put the alternative in seven words:

Kent C. Dodds built the Testing Trophy on that line. Agent-written code breaks the pyramid for a different reason, an assumption nobody wrote down: that each unit test was written by someone who knew what the unit was for.

When a person writes applyDiscount and its test, the test catches the times their fingers disagreed with their head. When an agent writes both from one reading of "10% off orders over 100", the misreading goes into both. The code uses >, the test checks 150.00, and nobody asks about exactly 100.00. The wide base of the pyramid becomes a mirror.

Keep the unit tests; they are cheap and they catch slips. Just stop counting them as evidence when one author wrote both sides. For agent-written code, weight the suite by how independent each check is from the code's author. The checks worth most are written from the requirement and assert on outcomes the code cannot fake: the response, the stored row, the event that fired.

Automated testing vs test automation vs verification

The three terms are used interchangeably in job ads and vendor pages. They name different things.

Test automation is the plumbing; automated testing is what flows through it. Verification is a different question, and it is the one a coding agent's summary claims to have answered. The AI verification post defines it in full, and verification vs validation separates it from the older "did we build the right thing". What each one costs per commit, and when to pay for which, is in verification vs automation.

Nine test types and what each can prove

Every type asserts on something specific. What it can prove stops exactly where its evidence stops.

Read the last column first when you choose automated testing tools. Each one covers its type's middle column for the paths someone asserted on, and nothing else. A suite with all nine still says nothing about a behaviour nobody wrote a check for. Exploratory testing is the only row designed to find those, which is why it resists automation the most.

The judgement gap

A green suite is a claim about assertions, not about the product. It tells you every expectation someone wrote held on this run. It says nothing about the expectations nobody wrote, and an agent that writes tests from its own code writes the expectations that code already meets.

The deciding part has a name, the test oracle. The AI test automation guide covers why generating the oracle with the same model moves the problem instead of solving it, so this guide will not argue it again. The practical version is what a false green looks like: a "Saved" toast, a passing test that checked for it, and a save request that returned 500.

Three questions tell you how much a green suite is worth:

1. What does it assert on? What the page showed, or what the system did?
2. Who chose the expected values? The requirement, or the code's current output?
3. Would it fail if the feature were broken? Break the feature on purpose and see.

The cost curve

Automated testing has four costs, and they move in different directions.

Authoring has fallen close to zero. A generator writes a plausible test in seconds, which is why suites now grow faster than anyone reads them.

Running is flat per run. A compiled suite (Jest, Playwright, pytest) spends CI minutes and no model tokens, and nothing beats that for a check that never changes.

Maintaining grows with suite size and UI churn. Flaky tests and self-healing tests are the two ways it shows up, covered in flaky tests and self-healing tests.

Re-running compounds, because every commit multiplies it. It bites hardest when a model drives the browser on each run, which is how agent-based testing automation tools re-check a flow.

In the published benchmark summarised in this comparison of regression tools, replaying four saved flows cost about 480 tokens. Re-driving them with an LLM cost about 120,000 tokens through Playwright MCP and about 129,000 through Chrome DevTools MCP, roughly 250 times more.

The shape to remember: the cheap part got cheaper, and the expensive part moved to judgement and re-runs. A plan that only measures authoring speed is measuring the cost that no longer matters.

Seven principles for a suite that stays honest

1. Write the expected value from the requirement, never from the output.
2. Assert on what the system did, such as responses, stored state and events, not only on what it showed.
3. Watch every new test fail once before you trust it.
4. Keep the author of a check independent of the author of the code where you can.
5. Treat a healed or re-recorded test as a changed test, and review it like one.
6. Give each stage a time budget, and move checks that blow it to a later stage.
7. Delete any test that cannot fail.

A five-level maturity model

Level 1: scripted manual. People follow written test cases before a release. Coverage is whatever fits in the afternoon, and nothing runs on a commit.

Level 2: automated execution. A suite runs in CI on every push and blocks merges on red. The green is trusted because a person wrote each test, which holds until people stop writing them.

Level 3: layered. A small commit gate, a slower regression run, contract tests between services, and flakes tracked with owners. The suite is fast enough that nobody skips it.

Level 4: independent judgement. Expected values come from requirements, new tests are proven to fail, and healed or regenerated tests are reviewed. Agent-written tests are treated as drafts.

Level 5: verified outcomes. Every change, including an agent's, is checked against what the running system did: requests, state and errors, with the evidence attached to the verdict. Re-runs are cheap enough to happen on every commit without a model in the loop.

A team that adopts a coding agent at level 2 or 3 drifts toward level 2's weakness: more tests, written by something that never knew the requirement. Level 4 does not happen by accident.

Glossary

• Test harness: the code and data that run tests automatically, from inputs and stubs through to the pass or fail record (test harness).
• Smoke test: a small, fast check of critical paths that decides whether a build is worth testing further (smoke testing).
• Regression test: a check re-run after a change to confirm that behaviour which used to work still does (regression testing tools).
• Flaky test: a test that passes and fails on the same code because it depends on something that varies between runs (flaky tests).
• Self-healing test: a test whose runner repairs it when it breaks, so it agrees with the app whether or not the app is right (self-healing tests).
• Test oracle: whatever decides if an observed result is correct, usually an expected value someone wrote (AI test automation).
• False green: a passing check on a broken app, because it asserted on something that looked right (why a green check isn't proof).
• Exploratory testing: learning the product, designing tests and running them at the same time, each test chosen from what the last one showed (exploratory testing).
• AI verification: checking whether a change an agent made produced the outcome it claimed, by reading the running app (AI verification).
• Agentic testing: either an agent that tests your app, or tests that check an agent by its tool calls and side effects (agentic testing).
• Verification vs validation: verification asks whether it was built right; validation asks whether the right thing was built (verification vs validation).
• AI test case generation: a language model proposing inputs, steps and expected results from a prompt, a spec, recorded flows or past bugs (AI test case generation).

Fun fact

The test pyramid comes from Mike Cohn's 2009 book Succeeding with Agile, where it is the "Test Automation Pyramid". Cohn first drew it with Lisa Crispin in 2003 or 2004, and "Jason Huggins independently came up with the same idea around 2006" (Fowler). Huggins had started Selenium in 2004 (Selenium history), a tool built for the very top of that pyramid.

Conclusion

Automated testing repeats decisions; it does not make them. Execution is solved. Authoring and maintenance are mostly automated, and judgement is automated only up to the comparison. When an agent writes the code, the decision about what correct means is the part to protect: write it from the requirement, check outcomes rather than appearances, and make re-runs cheap enough to run on every commit.

Reticle is a tool for that last step on web apps. An SDK in your development build records each step's requests, state and console, and saved flows replay as checks with no model in the loop. Its published scorecard is the source of the 480-token figure above, and records 0% flake across eight replays of one flow.

FAQ

What is automated testing?

Automated testing is software running checks that a person specified in advance and reporting whether each held. It repeats decisions cheaply and at scale, on every commit. It cannot decide what correct means: a suite only checks the expectations someone wrote, so a green result proves those held, not that the product works.

What is the difference between automated testing and test automation?

Automated testing is the practice of having software run your checks. Test automation is the engineering that makes the practice work: test automation frameworks, harnesses, pipelines, test data and upkeep. You do automated testing; you build test automation. Neither one is verification, which asks whether a specific change did what its author claimed.

What are automation testing tools?

Automation testing tools are the software that writes, runs, judges or maintains automated checks. They include unit runners like Jest and pytest, browser drivers like Playwright and Selenium, contract tools like Pact, and visual comparison services. Choose them by what each can observe, because a tool can only fail on evidence it actually reads.

What is an automated test system?

An automated test system is the full setup that runs checks without a person: the test cases, the harness that runs them, the environments and data they need, the CI pipeline that triggers them, and the reports. The cases decide what is checked; the rest of the system decides how fast and how reliably.

Does automated testing replace manual testing?

No. It replaces repeated manual checking of known behaviour, which machines do faster and more consistently. It does not replace deciding what correct means, or exploring for problems nobody anticipated: those stay human, or move to an agent whose findings a person reviews. Teams that automate well usually spend their manual time on exploration.