Teardown: Midscene.js passed a failed save 10 times out of 10

Midscene.js is an MIT-licensed test framework that "brings AI vision to E2E testing": you describe a check in plain language and a vision model reads a screenshot to decide it. Version 1.17.1 shipped on October 26. We ran its aiAssert 60 times against four bugs. Our view: it is a dependable check of what a user can see, and it should never be asked about anything else, because when we did, it answered with confidence instead of saying it could not tell.

What Midscene claims

The README is clear about the mechanism. Midscene models assertions "on how people use software: look at the screen, act on what you see, and check the visible result." It works on <canvas> and native apps, where selectors fail, and it lets you choose the model.

That is a fair description, and nothing below contradicts it. The question for a team adopting it is narrower: what happens when an assertion is about something the screen does not show?

What we ran

We wrote a 28-line local app: a profile form with a Save button and a small cart. Save shows a green "Saved" toast as soon as it is clicked, before the request returns, which is a common optimistic-UI pattern. The server runs one of four cases:

Playwright clicked Save, waited 600 ms, and handed the page to Midscene's PlaywrightAgent. Midscene only judged. We used @midscene/web 1.17.1 with gpt-4.1 as the vision model, ran three assertions five times on each case, and logged the real response statuses beside every verdict.

await page.click('#save');
await page.waitForTimeout(600);
const agent = new PlaywrightAgent(page);
await agent.aiAssert('The profile was saved exactly once and no error occurred');

Results

Sixty verdicts, and not one disagreement between runs of the same case:

The visible bug was caught every time. Midscene read $56 against $40 and $25 and failed it in five runs out of five. A selector-based assertion would have needed someone to think of checking the arithmetic.

The two invisible bugs passed every time, including under the third assertion, which names them outright. Our runner's log showed a 500 on every silent-500 run and two 200s on every double-save run. Median time per aiAssert was about 1.6 seconds.

Asking it why

We then asked the same model, through Midscene's aiAsk, on the silent-500 page: "Was the profile saved exactly once, with no error?" It answered:

"yes, the green 'Saved' notification confirms the profile was saved exactly once with no error."

That sentence is the finding. The model cannot see the network, so the honest answer is "the page does not say". It gave a confident yes and cited the one element that was wrong. Any check that reads only the screen will do the same, because the toast is the only evidence it has.

What it verifies, and what it misses

Verifies: text, layout, colour and arithmetic that appear on screen, including content drawn on a canvas. In our run it was also consistent: same input, same verdict, 60 times.

Misses: anything that happens off screen. Failed requests behind an optimistic UI, duplicate submissions, writes that never persisted, console errors. It does not refuse to judge these. It passes them.

Caveats. One model, one app, four cases. Midscene's own examples use other models, and a different model might phrase things differently, but no model can read a response it was never shown. We wrote the bug to be invisible on purpose, and a human looking only at the screen would also have passed it.

How to use it well

Write every aiAssert as a claim about the screen, never about the system. "A green Saved toast is visible" is honest. "The profile was saved" is a claim the screenshot cannot support.

Then put the system claims where the evidence is. In Playwright, the save request itself is one line away (network docs):

const res = await page.waitForResponse((r) => r.url().includes('/api/save'));
expect(res.status()).toBe(200);

That check would have failed every silent-500 run. Counting requests to that URL would have caught the double save. Use Midscene for what the user sees and the network or app state for what the app did, which is the split we argue for in our comparison of AI testing tools.