On this page
Ask five teams what a verification layer is and you get five answers: a test suite, a dashboard, an eval score, a screenshot diff, a CI check. A term with no standard cannot be held to anything. This page gives it a working definition, and four criteria any product that claims it can be measured against.
What is a verification layer?
A verification layer is the system that runs AI verification on every change. AI verification is the step that checks, inside the running application, whether a change an AI agent made produced the outcome it claimed: the request succeeded, the state changed, nothing errored on the way, and the result traces back to the source line that caused it.
The second sentence is the definition from the AI verification guide, reused word for word. The layer is that step made permanent, run on every change whether or not anyone asks. It names an AI agent because that is where the gap opened, but a human's commit gets the same check.
Why the term appeared now
Agents author changes faster than a human can check them. A coding agent can open a pull request in the time it takes a reviewer to read the last one. At that rate review turns into sampling, and the unsampled changes are where regressions go.
A test asserts on what it was told to assert on, not on what the program did. If the assertion checks the status code, a 200 carrying the wrong quantity passes. When the same agent writes the code and the test, both share one blind spot. That is verification vs testing in one line: a test answers the question it was given, and verification asks what actually happened.
So the question moved from "did the tests pass?" to "did the program do what the change claimed?" Answering it takes something that reads the running system on every change, cheaply enough that nobody turns it off.
The four criteria
Each criterion is necessary. A product that misses one may still be useful, but it is doing something else.
The criteria are implementation-agnostic: they say nothing about how a product meets them or whether a model is involved. Any product that claims the term can be measured against them, including ones that do not exist yet.
1. It reads program truth
It reads four signals after an action: the DOM outcome, the network result, the application state, and the console. Rendered output alone is not enough.
This is necessary because each signal can be right while another is wrong. A page can show "Order confirmed" while the request sent a quantity of 1 instead of 3. A list can look correct while the store behind it holds stale rows that the next screen will read.
What fails it: anything that judges a screenshot alone, or a log alone. A pixel-perfect page over a failed write passes a screenshot check. The screen is the last place a bug shows, and often where it hides.
2. It returns a verdict, not a report
It answers pass or fail, and gives the same answer twice for the same code and inputs.
A gate can only act on a verdict. A score of 0.82, or a page of observations, hands the decision back to a person, the bottleneck the layer exists to remove. Reproducibility matters as much: if two runs on one commit disagree, the red one gets ignored, and soon so do the rest.
What fails it: a judgement sampled fresh on every run, such as a model asked "does this look right?" with no fixed check. So do a dashboard you have to interpret and a flaky check retried until it turns green.
3. It is independent of the actor that made the change
The actor that made a change cannot rule on whether it worked. Separate the expectation, what should be true, from the verdict, whether it is. The verdict must come from the running system through a fixed check, not the author's report. The author may draft the expectation, but it must be fixed before the change and reviewed by someone else.
An agent that misread the ticket drafts an expectation for its misreading, and the check passes. Safety-critical engineering agrees: NASA's IV&V overview notes that IEEE defines independence as technical, managerial and financial.
What fails it: an agent reporting "tests pass" on checks it wrote and nobody else read.
4. It binds a failure to a location a fix can start from
A failure arrives with an address: a file and line, or the exact state transition that went wrong.
A red result with no address is a report, not a verification. The fix starts with a search, and for a coding agent that is the difference between a targeted edit and a guess.
Compare two failures for one bug. One says "checkout test failed". The other says the cart's order quantity is 1, expected 3, and the request came from the order handler at a named file and line.
What fails it: an end-to-end test that reports only a selector timeout and a screenshot. It says something broke, not where to start.
What a verification layer is not
Five adjacent categories get confused with this one. Each is useful, and none meets all four criteria on its own.
| Adjacent category | What it actually does | Why it is not this | What it is still useful for |
|---|---|---|---|
| Test framework | Runs the assertions someone wrote and reports which failed | Checks only what it was told to (criterion 1), often written by the change's author (criterion 3) | Fast unit and contract checks, the base of any suite |
| Observability platform | Collects logs, metrics and traces from production | Reports after users hit the problem, as dashboards and alerts rather than a verdict on one change (criterion 2) | Finding what broke in production, and how often |
| LLM eval harness | Scores model outputs against datasets or rubrics | Returns a score over samples, often graded by a model (criterion 2), and measures a model rather than your app's state | Choosing and regression-testing prompts and models |
| Visual regression tool | Compares screenshots with a baseline | Reads rendered pixels only (criterion 1): a correct screenshot over a failed write passes | Layout and styling regressions the DOM cannot show |
| CI gate | Blocks a merge when a configured check fails | Has no judgement of its own; it is exactly as good as the check it runs | The place a verification layer's verdict should land |
How to test the claim
Whether a product calls itself an AI verification platform or an AI verification tool, five questions separate the claim from the label. Each works on any product, your own suite included.
1. Break the request, keep the screen: does it fail? Make the page render success while the request sends a wrong value. If the run passes, the product reads rendered output, not program truth.
2. Run it ten times on one commit: is it the same answer? Compare ten verdicts on an unchanged build. Any disagreement means the verdict is sampled, and a sampled verdict cannot gate a merge: nobody can tell a real red from noise.
3. Who writes the check, and who rules on it? If the verdict is the model's own judgement of its own change, independence is missing. If the agent drafts the expectation, ask whether a person reviews it before it gates a merge.
4. Plant a bug: where does the red point? Hard-code one wrong value in a handler. You want a file and line, or the wrong state value next to the expected one. A timeout plus a screenshot is a report.
5. What does one re-run cost, measured? Ask for tokens, minutes or money per re-run of a real suite, measured rather than estimated, and whether teams actually run it on every commit.
The cost condition
A check too expensive to run on every commit becomes a check that runs on Fridays. Agent changes arrive continuously, so an occasional check covers only a sample. Verification counts only if it is cheap enough for every change; the per-commit math is in verification vs automation.
The expensive way is a model re-driving every flow on every run: open the app, click through, judge it again. That costs tokens per flow per run, and brings sampling back into the verdict, which breaks criterion two.
The cheap way is to pay the model once, when a flow is first driven, then replay the recorded check with no model. One commit gets one answer.
Reticle is one implementation built against these criteria: it reads DOM, network, state and console, and replays saved flows with no model. Its published benchmark puts a four-flow re-check at about 480 tokens, roughly 250 times cheaper than an LLM re-drive (a compiled suite costs none), with 0% flake over eight replays. Criterion three is partly on you: the agent that makes a change also drafts a flow's expected outcome, so review that file before it gates anything. The AI verification guide covers the method.
Where it does not help
Pixel regressions. A button 4px out of line or the wrong font weight: DOM, state and network all agree, so every criterion passes. Run a visual regression tool alongside.
Cross-browser rendering. A layer verifies behaviour in the engine it runs in. If another engine lays out a flex row differently, the verdict will not see it. A browser grid answers that.
Sites you do not own. Without the build, a layer can still drive the page and read the DOM, network and console. State and file-and-line attribution need instrumentation compiled into the app, so criterion four mostly goes, and there is no commit to gate.
Uneven framework coverage. Reading state and mapping an element to its source line depend on per-framework hooks. Expect the full set on some frameworks and less on others, and ask which is which.
A wrong expectation. If the check encodes the wrong requirement, the layer proves the wrong thing, precisely. Deciding what should be true is still a person's job, which is why verification vs validation stays a separate question. Keeping the checks from rotting as the app changes is the subject of AI test automation.
Fun fact
Before John Glenn's 1962 orbital flight, IBM computers had been programmed with the equations for his trajectory. Glenn asked engineers to "get the girl", the mathematician Katherine Johnson, to run the same numbers by hand on her desk calculator. "If she says they're good," she remembered him saying, "then I'm ready to go" (NASA). NASA's independent verification facility now carries her name.
Conclusion
A verification layer is AI verification made permanent: the check that runs on every change and answers with a verdict. The four criteria are the test of the term: program truth, a reproducible verdict, independence from the author, and a failure with an address.
Take the five questions to whatever you run today and count which criteria it meets.
FAQ
What is a verification layer?
A verification layer is the system that runs AI verification on every change, checking inside the running application whether a change produced the outcome it claimed. It reads the DOM, network, state and console, returns a reproducible pass or fail that never rests on the author's own report, and points each failure at a file, line or state transition.
Is a verification layer the same as a test framework?
No, a verification layer is not a test framework, although it can run on top of one. A test framework executes the assertions someone wrote and reports which failed. A verification layer decides whether a change worked by reading the running program, independently of its author, and returns a verdict a merge gate can act on.
Do you need a verification layer if you already have tests?
You need a verification layer if code reaches main faster than people can check it, which is common once agents write changes. Tests check what they were told to check, and are often written by the change's own author. The layer covers the gap: what the program actually did after the change, checked on every commit.
How do you evaluate a verification layer?
Evaluate a verification layer with five checks on a trial: break the request while the screen still shows success, run one check ten times on the same commit, ask who rules on pass or fail, plant a bug and see where the failure points, and measure what one re-run costs on every commit.
