Agentic testing: two meanings, one that survives

Search for agentic testing and almost every result describes the same thing: an AI agent that drives your app and writes your tests. That is one meaning. The other runs the opposite way: tests that check an agent. Teams that mix the two up buy a test-writing agent when what they needed was a way to check one.

Both meanings sit under AI verification, the step that decides whether a change did what its author says. This post defines both and argues the second is the one that lasts.

What is agentic testing?

Agentic testing means one of two things. Agent-as-tester: an AI agent explores an application, writes and runs tests, and repairs them when the UI changes. Agent-as-subject: tests that check an AI agent by its tool calls, side effects and resulting state. The first finds behaviour; only the second can tell you an agent did the right thing.

The rest of this page uses those two names. Most vendor pages describe only the first, which is why the second keeps getting bought by accident, or not at all.

Two meanings, one confusion

The confusion is expensive because the two need different tools. A test-writing agent cannot check a support agent's refunds, and a refund checker will not write your checkout tests.

Agent as tester: good at breadth, blind on correctness

Agentic AI test automation earns its keep in three places.

Breadth. An agent will try the journeys nobody scripted: the back button mid-checkout, a coupon applied twice, a currency switch after the cart is full. It is a tireless junior tester for exploratory testing.

Generation. Point it at a ticket and a running app and it drafts the test cases and the code to run them. Most of the draft is usable, and the rest is cheap to delete. The code an agent writes fails in its own repeatable ways, covered in testing AI-generated code.

Recovery from brittle selectors. When a button's ID changes, a scripted test fails. An agent looking for "the Pay button" finds it anyway, which removes a whole class of maintenance.

Where it stops is the outcome. The agent can reach the confirmation page, but it cannot know whether "Order placed" was the right result, or whether the charge behind it fired twice. If the same model decides pass or fail, it grades its own homework, and the gap between "looks done" and "is done" is exactly where it is weakest.

Agent as subject: what testing an agent actually requires

An agentic test in the second sense checks what the agent did, in four layers:

1. Tool calls. Which tools it called, with which arguments, how many times. Our MCP testing guide has a runnable harness for this layer.
2. Side effects. What changed outside the agent: the refund issued, the row written, the email sent.
3. Intermediate state. What the agent believed between steps: the order it looked up, the plan it formed.
4. The final message. What it told the user.

The final message is the least reliable thing to assert on. It is generated text, written to sound finished, and it can be fluent and wrong at the same time. Here is a recorded run of a support agent, tested on what it did:

// agent-test.mjs: test a support agent by what it did, not what it said. Node 18+, no dependencies.
import assert from "node:assert/strict";

// A recorded run: the agent's tool calls, and the final message it sent the user.
const run = {
  calls: [
    { tool: "lookup_order", args: { id: "A-1042" }, result: { total_minor: 499900, currency: "INR" } }, // ₹4,999.00 in paise
    { tool: "issue_refund", args: { order: "A-1042", amount_minor: 4999 }, result: { ok: true } },
  ],
  final: "Done! I've refunded your full order of ₹4,999.",
};

const check = (name, fn) => {
  try { fn(); console.log(`PASS  ${name}`); }
  catch (err) { console.log(`FAIL  ${name}: ${err.message.split("\n")[0]}`); process.exitCode = 1; }
};

check("final message claims a refund", () => assert.match(run.final, /refunded/));
check("refund was issued exactly once", () =>
  assert.equal(run.calls.filter((c) => c.tool === "issue_refund").length, 1));
check("refund amount equals the order total", () => {
  const order = run.calls.find((c) => c.tool === "lookup_order").result;
  const refund = run.calls.find((c) => c.tool === "issue_refund").args;
  assert.equal(refund.amount_minor, order.total_minor,
    `refunded ${refund.amount_minor}, order total ${order.total_minor}`);
});

$ node agent-test.mjs
PASS  final message claims a refund
PASS  refund was issued exactly once
FAIL  refund amount equals the order total: refunded 4999, order total 499900

The message is cheerful and the refund is one hundredth of the order. A test that read only the reply would have passed it. The bug lives in an argument, a unit mix-up between rupees and paise, and only a check on the tool call can see it.

The determinism problem

This is where the two meanings meet, and where most agentic AI testing goes wrong.

An agent re-driving the same flow takes a different path each run. It clicks a different element first, waits a different amount, phrases its reasoning differently. So when a run goes red, the cause may be the test, not the product. You have built a flaky suite on purpose.

The fix is to use the model once. Let the agent explore and find a path, record that path with its evidence, and replay it on every later run with no model in the loop. A replay cannot wander, so a red result means the product changed.

It also changes the economics. In our own published benchmark (the scorecard, 2026-10-06), re-checking a four-flow suite by LLM re-drive costs roughly 120,000 tokens, about 30,000 per flow. Replaying the recorded paths costs about 480, roughly 251 times fewer. That is the difference between checking every commit and checking when someone remembers.

Where MCP fits

The Model Context Protocol gives agents reach: a server exposes tools, and an agent calls them. Every tool an agent can call is a side effect someone has to test, so MCP makes the agent-as-subject meaning more urgent, not less. The protocol specification defines how tools are described and called, which is also what a test needs to record. Testing MCP servers and the agents that use them gets its own guide.

The reference loop: explore, record, replay, verdict

1. Explore. An agent drives the running app, or the agent under test runs against a real task. This is the only step that needs a model, and it is where breadth comes from.

2. Record. Keep the path and what happened along it: the requests, the tool calls and their arguments, the state after each step. A recording without evidence is a transcript, and a transcript cannot be asserted on. The testing harness that replays it decides what it can catch.

3. Replay. Re-run the recorded path on every commit, with no model making choices. Same steps, same order, so differences come from the product.

4. Verdict. Decide pass or fail from what the system did: the charge fired once, the refund matched the order, the console stayed clean. A person decides what correct means; the check applies it the same way every run.

When agentic testing is the wrong tool

Exploratory work where no stable flow exists yet. A feature that changes daily has nothing to record. Explore by hand or with an agent, but do not build a regression loop on sand.

Purely visual regressions. If the question is "did the button turn the wrong shade", a screenshot diff answers it better than any check on state.

Anything you cannot run twice. A flow that sends a real wire transfer, or an agent action with an irreversible side effect, cannot be replayed safely. Stub the boundary first, or test it once by hand with a person watching.

Unit-level logic. A pure function with clear inputs and outputs needs an ordinary unit test. An agent adds cost and nothing else.

What this looks like in practice

Reticle is a reference implementation of the loop. An agent explores the running app; the path is saved as a flow with network, state and console evidence; replays run with no model, so eight replays of one flow gave one verdict, 0% flake.

In the benchmark it caught 85 of 88 seeded bugs with no false alarms. Re-running a four-flow suite costs about 480 tokens. The definition behind the verdict is in the AI verification guide.

Fun fact

The first famous test of a machine graded it on its final message alone. In Alan Turing's 1950 imitation game, an interrogator types questions and judges only the replies.

Turing predicted that within about fifty years, an average interrogator would have no more than a 70% chance of picking the machine after five minutes of questioning (Turing test). It is a test of how convincing the transcript is, which is precisely the property an agent's final message is optimised for.

Conclusion

Agentic testing names two opposite jobs. Agentic test automation brings breadth and cheap test generation, but cannot judge outcomes. Testing agents as subjects means checking tool calls, side effects and state, never just the reply.

Both converge on the same loop: let a model explore once, record the evidence, replay without the model, and decide from state. That is the version that survives contact with a real product.

FAQ

What is agentic testing?

Agentic testing has two meanings. In one, an AI agent explores an application, writes and runs tests, and repairs them when the UI changes. In the other, tests check an AI agent by its tool calls, side effects and state. The first finds behaviour; the second tells you whether an agent did the right thing.

What is the difference between agentic testing and test automation?

Test automation runs scripted steps the same way every time. Agentic testing uses an agent that decides its own steps toward a goal, which adds breadth and resilience to UI changes but makes runs vary. The durable setup combines them: an agent explores once, and the recorded path is replayed as ordinary automation.

Is agentic testing the same as AI testing?

No. AI testing usually means either using AI to test software or testing AI models for accuracy and bias. Agentic testing is narrower: it involves agents, which act through tools and change state. Testing an agent means checking those actions and their effects, which model evaluation alone does not cover.

What is an agentic test?

An agentic test is a test whose subject is an agent. It gives the agent a task, records its tool calls and their arguments, and asserts on side effects and resulting state, such as the refund issued or the row written. It treats the agent's final message as a claim to check, not as the result.