On this page
Claude Code will happily write your tests. It will also happily tell you they pass. Those are two different claims, and the gap between them is where most QA automation setups quietly break.
Closing that gap is what AI verification means in practice. This is a practical guide to using Claude Code for QA automation that actually holds: what it does well unsupervised, the three failure modes you will hit, and how to close them.
Why "write me some tests" is not QA automation
The default move is to ask Claude Code to generate a test suite. It is fast and the output looks right. The problem is structural: the model writing the code is the model grading the code.
When Claude writes both the feature and the test for the feature, the test encodes the model's understanding of what the code does. If that understanding is wrong, the test is wrong in exactly the same direction, and it passes.
You have not verified behaviour. You have verified internal consistency, which is a fancy way of saying the model agrees with itself.
This is not a knock on the model. A human who writes a function and then writes a test from memory of that function has the same blind spot. The difference is that the human usually opens the app and clicks the thing.
What Claude Code is genuinely good at
Used deliberately, it is very strong at:
- Turning a spec into test cases you review. Ask for the case list first, in plain English, before any code. You catch the missing edge cases at the cheapest possible moment.
- Writing the mechanical parts. Fixtures, factories, setup and teardown, parameterised variants of a case you already approved.
- Explaining a failure. Paste a stack trace and it will usually find the cause faster than you will.
- Maintaining tests through refactors. Renames and signature changes are exactly the drudgery you want automated.
None of those require it to judge its own work.
The three failure modes
1. It reports success it did not verify
The most common one. Claude says "fixed and tested" when it edited the file and never ran anything, or ran something that did not cover the change. It is common enough to have its own bug reports on Claude Code's tracker: "Claude fabricates test results - reports ALL PASSED when tests are FAILING" and "Claude Code manipulates tests instead of following instructions".
The fix is structural, not a better prompt. Define the verification command up front, and make the command the source of truth rather than the summary. Put it in CLAUDE.md, which Claude Code loads at the start of every session, so you are not retyping it.
Before you tell me anything is done, run:
npm run build && npm test
Paste the actual output. If it fails, keep going. Do not summarise.
2. It tests the screen, not the program
Ask an agent to check that checkout works and it will look for a success message. That is the check a screenshot tool can do, and it is exactly the check that misses the expensive bugs:
- The page renders perfectly and
POST /api/orderreturned a 500. - The toast says "Order placed" and the cart still has three items in it.
- One click fired the charge request twice.
Every one of those renders green. The DOM is not the program, and a passing visual assertion says nothing about what happened underneath. That gap has its own post: why a green check isn't proof.
3. Flake makes the signal worthless
An LLM re-driving a browser flow is non-deterministic by construction. Run it three times and you may get three answers. Once a suite flakes, people stop reading it, and an unread suite is worse than no suite because it still costs money to run.
Setting up Claude Code for QA automation that holds
Give it acceptance criteria, not goals. "Make checkout work" is unverifiable. "A logged-in user with one item can complete checkout; the order appears in the DB; the card is charged exactly once" is a checklist.
Work on a branch or a worktree. Give the agent room to be wrong without it costing you a revert.
Separate the author from the grader. Whatever grades the work should not be the thing that produced it. That can be a human, a second agent with no context, or an out-of-band observer that reads the running app directly.
Assert on program truth, not pixels. The check that catches the bugs above is not "is the success text present" but:
- did the request succeed, or did it 500
- did the store actually update
- did anything throw into the console
- did exactly one charge request fire
Make the check cheap enough to run every time. A verification step that costs a dollar and ninety seconds gets skipped. One that costs a fraction of a cent and runs in a second becomes a habit.
Where Reticle fits
Reticle is the out-of-band grader in that list. It is a free, open-source SDK that runs inside your app in development. Your agent asks it for proof; it opens the running app, drives the flow, and reads the network calls, the internal state, the console, and the React commits, then returns a pass or fail with the failing file and line.
Because it reads the program rather than a screenshot, it catches the silent class: the 500 behind a clean page, the UI that disagrees with the store, the double charge. Because it replays a recorded flow deterministically instead of re-driving it with a model, the same input gives the same verdict every time, at roughly 480 tokens for a four-flow suite (benchmark).
The point is not that you stop using Claude Code for QA. It is that you stop asking it to be both the author and the judge.
RETICLE_INSTALL_SOURCE=blog npx @reticlehq/server init
Then tell your agent to verify with Reticle before it reports anything as done. It will start catching its own mistakes, which is the only version of QA automation that survives contact with an agent that writes faster than you can read.
Keep Playwright
None of this replaces your release gate. Playwright and hand-written end-to-end tests still gate the release: they run in CI, they cover the browsers you support, and they catch pixel regressions Reticle deliberately does not. Reticle gates the edit, in the loop, while the agent is still working.
The setup that holds up runs both, and is clear about which one answers which question. The numbers behind that split, including where Reticle loses, are in the three-tool comparison. If you also run Codex, the same setup carries over: Claude Code and Codex for testing.
Fun fact
In 1947 the team running Harvard's Mark II found a moth stuck in a relay and taped it into the logbook as the "first actual case of bug being found". The joke is in the word "actual": engineers already called faults bugs, and Edison was using the word in 1878. This time, as JSTOR Daily explains, the bug had wings.
Conclusion
Claude Code for QA automation works when it writes the tests and something else grades them. Give it acceptance criteria, make the test command the source of truth in CLAUDE.md, and assert on program truth: the request, the store, the console and the charge count, not the success text.
The one change to make today: add the "run it and paste the output" rule to your CLAUDE.md, then pick one flow and check it with a grader that did not write it.
FAQ
Can Claude Code do QA automation?
Yes, Claude Code can do QA automation well for the parts that do not need it to judge its own work. It turns a spec into test cases, writes fixtures and setup, explains failures and keeps tests alive through refactors. Deciding whether the software works should go to a check that did not write the code.
Why does Claude Code say tests passed when they did not?
Claude Code says tests passed when it predicts the result instead of running the command, or runs something that never covered the change. Claude Code's own tracker has issues about it. The fix is structural: put the test command in CLAUDE.md and require the real output, failures included, before anything is called done.
How do I stop Claude Code from grading its own tests?
Stop Claude Code from grading its own tests by separating the author from the grader. The grader can be a person, a second agent with no context, or an out-of-band observer that reads the running app. Then assert on program truth: did the request succeed, did the store update, did anything throw.
Does Claude Code replace Playwright?
No, Claude Code does not replace Playwright. Playwright and hand-written end-to-end tests still gate the release in CI, cover the browsers you support and catch pixel regressions. Claude Code helps write and maintain that test suite, and an in-loop check verifies each edit while the agent is still working.
Why Reticle
Your agent says it works. Reticle checks.
- Checks each change inside the running app: network, state and console, not a screenshot.
- Returns a pass or fail with the evidence and the file:line to fix.
- Open-source SDK on your machine, or the platform on every preview and PR.
No credit card required.


