On this page
Verification vs automation sounds like a vocabulary argument until the CI invoice arrives. Most teams pay for one and believe they bought the other, and the gap shows up as a green pipeline over a broken feature.
Automation answers whether the steps you scripted ran and the checks you wrote passed. Verification answers whether the change did what it claimed, read from the running program. The first is a statement about the run. The second is a statement about the product.
Both are worth paying for. This page is about knowing which one you are paying for, what each costs per commit, and when one is enough. The field guide to automated testing covers the practice in depth; this page draws the boundary.
What automation actually answers
The old comparison, automated testing vs manual testing, was about who runs the steps: a person or a script. Automation won that argument for anything repeatable, and most guides to manual vs automated testing stop there. But a script, however good, can only report on what it was told to look at. A passing automated run tells you four things:
- The steps executed. Every click, request and function call in the script happened, in order, without throwing.
- The expected element appeared. A selector matched something on the page within the timeout.
- A value matched an assertion.
expect(total).toBe(42)held for the value the script read. - A duration stayed under a threshold. The page loaded, or the job finished, inside the budget.
Each of those is a statement about the run. None of them says the product works. If the assertion reads the toast and the bug is in the request behind it, all four hold and the customer is still charged twice.
So is automated testing verification? Only where the assertions happen to cover what the change was meant to do. That is a property of who wrote the assertions and what they knew, not of the automation.
What verification answers
Verification asks a different question: after this change, did the program do what the change claimed? It reads the program's own evidence, the DOM, the network, the application state and the console, and decides from that.
The definition and the four criteria a verification layer has to meet are on their own page, and this one does not restate them. What matters here is the practical difference in test automation vs verification. Automation starts from a script and asks whether it passed. Verification starts from a claim and asks whether it is true.
That difference grows when an agent writes both the change and the tests. The agent's script and the agent's code share one understanding of the ticket. If that understanding is wrong, automation confirms it faithfully, every run.
Verification vs automation: the cost structure
Verification costs more to build. That is the honest starting point.
| Cost to author | Cost to run | Cost to maintain | Cost when it fails | What grows with commit volume | |
|---|---|---|---|---|---|
| Automation (scripted suite) | Low and falling: a generator writes a plausible test in seconds | CI minutes, no model tokens | Grows with suite size and UI churn: selectors, waits, flaky retries | A red with a stack trace, or a silent green that ships the bug | CI minutes and flake triage |
| Verification | Higher: instrument the app so state and network are readable, declare each flow's expected outcome, review it | A replay: hundreds of tokens per suite. A model re-driving each flow: tens of thousands per flow | Expectations change when behaviour does; recorded steps re-resolve their targets | A red with an address: the file, line or state value that was wrong | Replay cost, or model tokens if every run re-drives |
Read the last two columns together. Automation is cheap until it fails silently, and then the cost lands in production, where it is the most expensive kind. Verification is dearer up front and cheaper per failure, because the failure arrives with somewhere to start.
The CI bill
Per commit, a compiled suite costs CI minutes and nothing else. Nothing in verification beats that, and teams should not pretend otherwise.
The bill changes shape when a model is in the loop. In the published benchmark summarised in this comparison of regression testing tools, re-checking a four-flow suite cost:
| How the suite is re-checked | Tokens per re-run |
|---|---|
| A model re-drives each flow through one browser-automation server | about 120,000 |
| A model re-drives each flow through a second browser-automation server | about 129,000 |
| Recorded flows replayed, no model in the loop | about 480 (483 measured) |
That is about 250 times cheaper per re-run (251 measured), with 0% flake across eight replays of the same flow, because no model samples a different path each time. A compiled suite costs zero tokens and is not on this table.
Now multiply by commits. At 50 agent commits a day, re-driving four flows costs about 6,000,000 tokens a day (50 × 120,000). Replaying them costs about 24,000 (50 × 480). The counts are ours; the per-run figures are the benchmark's.
The crossover sits close to the first re-run. Recording a flow takes one model drive. Every later check is a replay instead of a re-drive, so if the recording drive costs about what one re-drive does, the recorded suite has paid for itself by the second commit it checks. That is an estimate from the per-run figures, not a measured break-even.
The decision rule
Automate only when the code is logic with a spec and people review the changes. A pricing-rules package, a parser, a date library: inputs in, outputs out, no UI, every diff read by a human. Unit and property tests are the exact oracle, and a verification layer would add cost without adding a question.
Verify only when the product changes faster than anyone can maintain a script. An internal dashboard an agent edits ten times a day, with no suite and no time to write one: record the three or four flows people actually use and check every change against them. Add scripted tests once the churn settles.
Both when people pay you and agents write changes. Scripted tests for the logic and the contracts, verification on the money paths: sign-up, checkout, the thing that sends email. This is most teams with customers, and it is where the two stop competing.
Where automation plainly wins
Deterministic unit behaviour. A pure function with known inputs has an exact answer. A unit test is the cheapest, fastest, most precise oracle there is, and no layer that reads a running app improves on it.
Cheap regression on stable code. Code that has not changed in a year needs a check that costs nothing to run. A compiled suite runs in seconds with zero tokens, forever.
Breadth of inputs. Property-based tests and fuzzers try thousands of generated inputs per run. Verification drives a handful of real flows. For "does this hold for every input", automation is the right tool by a wide margin.
Anything that must run offline. Pre-commit hooks on a plane, air-gapped builds, embedded targets with no browser. A script needs a runtime; it does not need a network, a model or a running app.
None of this is a concession. It is the part of the job automation does better than anything else, and the reason the decision rule ends with "both". Keeping a scripted suite from rotting as agents edit it is its own discipline, covered in AI test automation.
Fun fact
Even an automated check can be switched off without anyone noticing. Python's own language reference says the code generator "emits no code for an assert statement when optimization is requested at compile time". Run the same file with python -O, and assert 1 == 2 stops failing: the steps run, the script exits cleanly, and nothing was checked.
Conclusion
Automation tells you the steps ran and the checks you wrote passed. Verification tells you the change did what it claimed. Pay for automation everywhere it is cheap and exact, and add verification where a silent green would cost you a customer.
Reticle is one tool built for the verification side: it records flows inside the running app and replays them with no model in the loop. The method is in the AI verification guide, and the per-run figures above come from its published benchmark.
FAQ
What is the difference between verification vs automation?
Verification vs automation is a difference in the question answered. Automation answers whether scripted steps ran and written checks passed, which is a statement about the run. Verification answers whether a change did what it claimed, read from the running program's DOM, network, state and console, which is a statement about the product.
Is automated testing the same as verification?
No, automated testing is not the same as verification, although it can verify where its assertions cover what a change was meant to do. Automated testing confirms the checks someone wrote. If those checks read the wrong signal, such as a toast instead of the request behind it, the run passes while the feature is broken.
What is the difference between manual and automated testing?
The difference between manual and automated testing is who runs the steps: a person or a script. Manual testing adapts and notices the unexpected but is slow and hard to repeat. Automated testing repeats exactly and cheaply, but only checks what it was told to. Neither, by itself, settles whether a change worked.
How much does test automation cost to maintain?
Test automation maintenance cost grows with suite size and with how often the interface changes. Selectors break, waits turn flaky, and retries hide real failures. Running a compiled suite costs only CI minutes, so maintenance, not execution, is usually the bigger line item, and it rises fastest on fast-changing UI code.
