AI QA tools: buy, build, or neither

Most teams choose AI QA tools from a demo. The demo shows a checkout flow turning green, and nobody in the room can see how the tool decided it was green. That decision is what you are buying, and a demo is built to hide it.

This page is for the person who signs the order. It gives you the eleven questions that separate a demo from a system of record, an honest build-versus-buy comparison, and a pilot that ends in a decision after two weeks.

If you are already comparing vendors, Talk to us and bring the questions below. To try it first, Start a 14-day trial (sign in, then add a card under Billing).

What the category covers

AI QA tools put a model somewhere in your quality process: writing tests, driving the app, or judging the result. Those are different jobs with different failure modes, and our AI QA guide separates them. This page is about buying.

AI QA tools: buy versus build economics

Every team that evaluates AI QA automation asks whether it could build the thing. Often it can build part of it. The table shows what each part costs to build in-house, what it will not do, and where building stops paying.

The effort column is our rough estimate for one experienced engineer, not a measurement. Use your own rates.

The first row is the one most teams should build, and many already have. The last two rows are where buying usually wins, because they are infrastructure nobody gets promoted for maintaining.

The eleven questions

Paste these into your RFP as written. A vendor worth shortlisting answers each in writing, with evidence, without a follow-up call. The first seven expand the short list in our AI QA guide.

1. Who or what decides pass or fail: a model, or a deterministic check? Ask for the exact step where the verdict is computed. If a model decides, ask which model and what it is shown.

2. Run the same build twice. Do you get the same verdict? Ask for their measured agreement rate on unchanged builds, and how many replays it is based on.

3. What does the verdict read? Pixels and page text, or the requests, application state and console output behind them? A check that reads only the screen cannot see a duplicate payment.

4. When it says pass, can I see the evidence? Ask for a sample pass record: what was checked, what was observed, and whether someone outside your team could replay it.

5. Where does the definition of correct live, and who edits it? In your repository under review, or in the vendor's dashboard? Can you diff it between releases?

6. What does the thousandth run cost? Ask for the marginal cost of re-checking a flow that already passed once. Per-commit testing makes this the number that matters.

7. What happens to our tests and history if we stop paying? Ask what you can export, in what format, and whether it runs without the vendor.

8. What leaves our machines, and what is retained? Ask for a list by data type: source code, DOM, request and response bodies, screenshots, credentials. For each, ask where it goes and for how long.

9. What does your SDK or agent do in a production build? Ask whether it is removed, disabled, or still present, and how they prove it.

10. What happens to our CI if your service is down? Ask whether licensing, test runs and verdicts depend on reaching the vendor, and what fails closed.

11. How do twenty red runs become one bug? Ask how failures are grouped, how flaky checks are marked, and whether the tool names the first commit that went red.

Verdict provenance: the only question that matters

Questions 1 to 3 tell you where a verdict comes from. Call it provenance. Every other answer depends on it.

If a model decides, the verdict inherits the model's habits: it can be persuaded, it varies between runs, and it tends to agree with the code it is shown. That is fine for drafting tests. It is a problem for deciding whether a release ships.

If a deterministic check decides, the verdict is repeatable and auditable, and the question moves to coverage: what the check reads. A check that reads only pixels will pass a page that looks right and charged the customer twice.

So the shortlist rule is simple. A tool whose answers to questions 1 to 3 are "a model", "usually" and "the screen" is a test generator or an exploration aid. Do not let it gate a release.

Data handling and residency

These AI quality assurance tools read your running application, which makes them a security review item. Require written answers to questions 8 to 10, and check them against the code where you can.

A buyer should require four things:

• A list of what leaves the machine, by data type, with a way to turn each off.
• A retention period for anything stored, and a way to delete it yourself.
• Proof the tool is absent from production builds, not a promise that it is idle there.
• Licensing that survives a vendor outage, so your CI does not stop when theirs does.

For transparency, here is Reticle's posture as stated on reticle.sh.

It sends no code, DOM, network traffic, console output or file paths. The SDK is dev and preview only, tree-shaken out of production builds, and its bridge binds to 127.0.0.1 behind a token. The CLI sends anonymous usage events, which reticle telemetry disable turns off.

License keys are signed with Ed25519 and verified offline. The SDK is Apache-2.0 and the server is source-available. If you link a project to Reticle Cloud, runs sync to your workspace, as the privacy policy describes.

Team workflow: what a tool must keep

A single red run is a question. Twenty red runs across a week are either one bug or noise, and the tool should tell you which. Google found that "about 84% of the transitions we observe from pass to fail involve a flaky test" (Google Testing Blog).

To turn twenty reds into one bug, a tool has to keep:

• History per flow, keyed by commit, so you can see the first run that went red.
• A failure signature: the step and the evidence that failed, so identical failures group together.
• A flake record for each check, so a known-unstable check is marked before anyone investigates it.
• The evidence itself, kept long enough to compare a failure with the last pass.

If a vendor's history is a list of runs with a status column, your team will do this grouping by hand, in a chat thread.

Pricing shapes and their traps

We will not quote prices: they change and depend on contracts. The shape of the price matters more than the number, because it decides what your team will avoid doing.

• Per seat. Cheap for a small team, but coding agents and CI are not seats. Watch for results that only seat holders can read, which keeps findings in one person's inbox.
• Per run. Fair on paper, but it charges you for checking more often. Teams on per-run pricing quietly stop running on every commit, which is when checks are worth most.
• Per test. It rewards giant tests that fail opaquely, and AI-generated suites inflate the count fast. Ask how a test is counted.
• Platform fee. Predictable, but usually with caps. Ask what happens at the cap: runs queue, runs stop, or overage is billed.

Model each shape against your real commit rate.

A two-week pilot design

A pilot should end in a decision. Give it fixed flows, seeded bugs, and numbers decided in advance.

Week one: measure. Pick five flows, including one that broke in production last year. Seed three regressions your current suite misses: one visual, one in a request, one in state. Then measure:

• how many seeded regressions each tool catches;
• whether ten replays of an unchanged build agree;
• how long it takes someone outside the pilot team to explain a red from the tool's evidence alone.

Week two: decide. Run the tool on real pull requests. Count true reds, false reds and false passes, and price the run volume you saw at your real commit rate. Then decide: buy, build the first row of the table, or neither.

Three numbers that end the pilot early:

1. One false pass on a seeded regression. A missed bug you planted is a missed bug in production.
2. Any disagreement between replays of the same build. That is noise you will pay for every day.
3. One red your team cannot explain from the evidence without reproducing it by hand. The tool has moved work, not removed it.

Seven red flags

• The demo shows only passing runs.
• "The AI decides" comes with no evidence attached to the verdict.
• Verdicts change between replays and the vendor calls it self-healing.
• Pricing is per test and the tool writes the tests.
• You cannot export tests or history in a format that runs without the vendor.
• The pilot requires production credentials or a production SDK.
• Security answers come as a slide deck rather than as code or documents you can check.

The QA agents AI vendors now sell are often good at exploring an app. Hold them to the same eleven questions, because an AI QA agent that explores well can still judge badly. For the deterministic side, compare regression testing tools on what they read, and see why AI test automation tends to rot when nobody owns the verdict. When you shortlist QA agent tools, start with question 1.

Our best AI testing tools list shows each one's verdict source.

Fun fact

In 1996 Ariane 5's first flight failed 37 seconds after ignition. The inquiry board found that the inertial reference system's alignment function "was not taken into account in the simulations" and that the equipment and system tests "were not sufficiently representative" (ESA). A demo is the software version of an unrepresentative test.

Conclusion

Buying AI QA tools is mostly a question of verdict provenance. Ask who decides pass or fail, whether the answer repeats, and what it reads. Build the scripted suite yourself, buy the evidence and history layers if you need them, and run a two-week pilot with stop conditions agreed before it starts.

FAQ

What are AI QA tools?

AI QA tools use machine learning models, usually large language models, inside software quality assurance. They generate tests, drive an application, triage failures or judge results. They differ most in what decides pass or fail: a model, or a deterministic check that reads requests, state and console output. That difference decides whether a tool can gate a release.

Are AI QA tools worth it?

They are worth it when the verdict is repeatable and comes with evidence your team can read. They are not worth it when a model decides pass or fail with nothing attached, because you trade writing tests for checking the tool. A two-week pilot with seeded bugs answers the question for your codebase.

How much do AI QA tools cost?

Prices vary by vendor and contract, so compare pricing shapes instead: per seat, per run, per test, or a platform fee. Each changes behaviour. Per-run pricing discourages checking every commit, and per-test pricing rewards large tests. Model the cost at your real commit rate, and ask what the thousandth re-check of a passing flow costs.

Can you build an AI QA tool in house?

You can build parts of one. A scripted end-to-end suite in CI takes roughly one to two engineer-months, and most teams should own it. A prompted browser agent is quick to prototype but hard to make repeatable. Evidence capture, replay and run history take many months and an owner, which is where buying usually wins.