Exploratory testing, rebuilt for agents

Most testing assumes you know what to check before you start. Exploratory testing assumes you don't, and that you will find out by testing. That one assumption is why it has resisted automation for forty years, and why it is the one practice that got more interesting, not less, once code started writing itself.

What is exploratory testing?

The exploratory testing definition most practitioners would sign:

Exploratory testing is an approach to software testing in which the tester learns about the product, designs tests and runs them at the same time, using what each test reveals to choose the next one. The tester is responsible for the quality of that loop, not for executing a script someone else wrote.

What separates it from unscripted clicking is the charter: a short mission that says what you are exploring, with what resources, and what you want to find out.

Cem Kaner coined the term in the 1980s, and the Wikipedia summary still uses the shortest version: simultaneous learning, test design and test execution. Exploratory testing in software testing is not a phase. It sits next to scripted checks, and it finds the problems nobody wrote a check for, because nobody knew they were there.

Exploratory testing vs ad hoc testing

Most teams collapse these two, and both get a bad reputation for it. The ISTQB glossary defines ad hoc testing as informal testing performed without test analysis and test design. Exploration does analysis and design constantly. It just does them during the session instead of weeks before it.

The practical difference shows up a month later. Ask "did anyone test refunds over the weekend boundary?" An exploratory team opens the session notes. An ad hoc team asks around.

The loop: charter, session, notes, debrief

Exploratory testing in software development runs as a short loop. You can run one tomorrow with nothing but a timer and a text file.

1. Write the charter. One or two lines. "Explore the refund flow with partial refunds and mixed currencies, to find where the amount shown and the amount sent can disagree." It names a target, the resources (data, accounts, a build) and the information you want.

2. Run a timeboxed session. Close email and chat. Test against the charter, and follow anything off-charter that looks important, but note that you did.

3. Take notes as you go. What you tried, the data you used, what surprised you, every bug and every open question. Notes written afterwards are memories, and memories are optimistic.

4. Debrief. Ten minutes with a lead or a peer. What was covered, what was not, what got in the way, and what the next charter should be.

The loop matters more than any single session. A charter that turns up three bugs in the currency code usually earns a second charter aimed straight at the currency code.

Session-based test management in practice

Jonathan and James Bach formalised the loop in their 2000 article, Session-Based Test Management. They call a session "an uninterrupted block of reviewable, chartered test effort". Session-based testing is still the clearest way to make exploration something a lead can plan and report on.

How long a session runs. In their team, about ninety minutes, give or take. Closer to forty-five minutes counts as a short session and closer to two hours a long one. They expected about three sessions per tester on a normal day, because meetings and email eat the rest.

What the charter contains. A mission statement and the areas to be tested. Keep it narrow enough that a session can finish it, and broad enough to leave room to learn.

What the notes must capture. Their session sheet has the charter, the tester, the start time, data files, test notes, issues and bugs. It also has a rough split of the time into test design and execution, bug investigation, and setup, plus how much time went "on charter" versus "on opportunity".

What the debrief decides. They walk through the sheet with an agenda they call PROOF: Past, Results, Obstacles, Outlook, Feelings. The debrief accepts the report, coaches the tester, and decides the next charters. It is also where a lead learns that a session at 50% bug investigation means the product is slowing the testing down.

What changes when an agent does the exploring

Here is the part that changed. A coding agent that drives a running app does learn while testing: it reads the page, tries something, sees what happened, and picks its next move from the result. That is the exploratory loop. What the agent usually lacks is everything around the loop that made exploration trustworthy for humans: the charter, the notes and the debrief.

An agent that explores without those is just clicking, fast. It will hand you a transcript of every action it took and a confident summary. A transcript of clicks is not evidence. It says what the agent did, not what the app did in response.

Evidence is the consequence. The refund request fired once, not twice. The store holds the new balance, nothing was logged to the console, and the order is still there after a reload.

That is the level where Reticle works: it sits inside the running app in development and reads the network, the application state and the console while the agent explores.

Three things change when exploration happens that way.

The path is recorded. Every flow the agent drives is saved as a flow under .reticle/flows/, with the steps it took and the outcome it expected. That file plays the role of the session sheet, written while it happened instead of afterwards.

The path replays deterministically. A saved flow re-runs without a model in the loop. Replaying the same flow eight times in our benchmark gave one status and one verdict all eight times: 0% flake.

The outcome is diffed against the last known good run. Save a baseline when a flow is right, and later runs are compared against it, so a change shows up as a difference you can read rather than a feeling that something looks off. That is what AI verification means in practice: the check reads what happened, not what the agent claims happened.

Turning an exploration into a regression asset

The old cost of exploration was that its findings were perishable. A tester found the weekend-refund bug, it got fixed, and nothing stopped it coming back except someone remembering to look again.

A recorded exploration changes that. The flow that found the bug is committed with the fix, and from then on it runs without anyone asking (the verification harness tutorial walks through it step by step):

# replay every saved flow against the running app; exits non-zero if one fails
npx @reticlehq/server verify http://localhost:3000

In CI that is one step. It runs on every commit and fails the build when a flow that used to pass stops passing. The cost is small enough to run every time: about 480 tokens to re-verify a four-flow suite, roughly 251 times cheaper than having a model re-drive the same flows. That comparison is against an LLM re-driving the flows, not against a compiled test suite, which costs no tokens at all.

The human part of the session does not disappear. Someone still writes the charter, reads what the agent found, and decides which explorations are worth keeping. Exploratory testing tools that skip that step produce a pile of recordings nobody trusts.

A saved flow is also only as good as the test harness replaying it: what the harness asserts on decides what the flow can catch, and whether it becomes one of your flaky tests.

Journey coverage, not line coverage

Line coverage tells you which code ran. It cannot tell you whether a person could finish what they came to do. For exploration, measure journeys.

• Journeys exercised. Which user journeys have a charter, a session and a saved flow, and which have none. "Refunds: three sessions, two saved flows. Account deletion: never explored." That list is the plan for next week.
• Outcomes asserted. For each saved flow, what it checks at the end: a request count, a value in the store, a signal the app emits, a clean console. A flow that asserts nothing is a recording, not a check.
• States reached. Which states the exploration got the app into: an empty cart, an expired session, a failed payment, a refund larger than the remaining balance. The verdict also reports what a suite never reached, which is where the next charter comes from.

On seeded bugs, this way of checking caught 85 of 88 in our published benchmark, with no false alarms. Two of the other three are traps a correct tool must not flag. The third is a genuine miss: stale data inside an iframe. The benchmark scorecard has the method.

If you want to run this on your own app, start free on the hosted platform, or follow the open-source SDK install and keep everything on your machine.

Anti-patterns

"Exploratory" as an excuse for no plan. If nobody can say what the session was for, it was ad hoc testing with a better name. Write the charter, even if it is one line.

Sessions with no notes. A session with no notes cannot be debriefed, repeated or trusted. The same goes for an agent session with no saved flow.

Charters so broad they mean nothing. "Explore the app" is not a charter. "Explore checkout with a coupon and a gift card together" is. If a session cannot finish its charter, split the charter.

Treating a bug found as the only output. A session that finds nothing still produced coverage, notes and a better next charter. Counting only bugs teaches people to stop exploring the parts that work.

Expecting the agent to be curious for you. An agent explores the journeys you point it at, and replay re-checks what was already found. It does not replace human curiosity, and it will not find a bug nobody thought to look for. That part is still the job.

Fun fact

The 2000 article that made exploration manageable also warned what happens when managers skip the debriefs. Without them, the authors wrote, session sheets "degrade into something more like rambling ransom notes or vaguely technical haiku" (Bach, Session-Based Test Management). Anyone who has read an agent's run log has met the haiku.

Conclusion

Exploratory testing works because the tester learns while testing, inside a charter, with notes and a debrief that make the learning reviewable. Agents can now do the learning part. They still need the charter from you, and they need the notes and the replay from the tooling, or their exploration is a transcript nobody can check.

Pick one journey nobody has explored this quarter. Write a one-line charter, run a ninety-minute session with an agent driving, save what it found as a flow, and put that flow in CI.

FAQ

What is exploratory testing?

Exploratory testing is an approach where the tester learns about the product, designs tests and runs them at the same time, choosing each next test from what the last one revealed. It is guided by a charter and recorded in session notes, which is what separates it from random clicking. It finds problems nobody wrote a scripted check for.

What is the difference between exploratory testing and ad hoc testing?

Exploratory testing has a charter, notes and a debrief, so you can say afterwards what was covered and what was not. Ad hoc testing is informal testing without test analysis or design, and usually leaves no record. Both are unscripted, but only the exploratory kind is planned, reviewable and repeatable. The difference shows when someone asks what was tested.

What is session-based test management?

Session-based test management is a way to plan and report exploration in timeboxed sessions of about ninety minutes. Each session has a charter, produces a session sheet with notes, data, bugs and issues, and ends with a debrief. Jonathan and James Bach described it in 2000 to make exploration measurable without turning it into scripts.

Is exploratory testing manual testing?

It has mostly been manual, because it depends on a person learning while they test. It does not have to be. An agent can drive the exploration while a person writes the charter and reviews the findings, and recorded sessions can replay automatically. What stays human is deciding what is worth exploring and judging what was found.

How do you measure exploratory testing?

Measure it by journeys, not lines of code. Count which user journeys have a charter, a session and a saved flow, what each flow asserts at the end, and which states the app reached. Session sheets add time on charter, time investigating bugs and time lost to setup, which shows where the testing is being slowed down.