Who checks the tests the agent wrote? Four answers from HN

The tagline on the Ruby mutation tester mutant reads: "AI writes your code. AI writes your tests. But who tests the tests?" Between October 23 and 27, Hacker News gave four answers across four threads. Our view: only one of them can be checked by a machine, which is making every test prove it can fail. Reading every generated test does not scale, and trusting them is not testing. We ran the one tool that does this, and it showed the limit too: mutation evidence says what a test detects, not what it is for.

Where it started

On October 21 a Substack post, Unit Tests are not the enemy, pushed back on teams deleting test suites because agents can regenerate code. Its one-line thesis: "The tests were never the problem. Nobody reading them was." The HN thread was small, but the same question came up all week, most loudly under a busy post on software quality on October 27.

The four answers

Nobody reads them. AlienRobot put it most sharply: "Depending on machine-generated tests that nobody reads isn't better than having poorer coverage." Such a suite, they argued, only guarantees that the code returns true for the current implementation. rafaelRiv was blunter: "If the test pass, ship it." eviks added the incentive problem: coverage metrics get worse as evidence when "bad tests are easier to write".

Read them. tadfisher disagreed: "If LLMs are good enough to write code, they are good enough to write tests," and they can be "quite creative" about scenarios. The real risk, in their view, is agents "changing the invariants encoded by the tests". Their fix was to "read the code they produce, tests included", and they said it is "not a common thing" they have caught agents doing. That is honest, and it works only while a human can still read everything.

Make them fail first. chuckadams described a project where the agent "has to see a test for a new feature actually fail before it considers the test any good", and then "does a mutation test to ensure it flips back to red". They also reported the cost: the agent made the test framework "complicated, byzantine, and ultimately brittle" until they forced a cleanup, twice.

Keep the writers apart. seanmcdirmid is building a process where "the lib agent can write lib code, but cannot look at test code", the test agent cannot see the library, and an auditor that can see both assigns blame. They say it costs more tokens but gets "better results". They are still benchmarking, and the repository is unreleased, so treat the claim as a hypothesis for now.

A fifth comment marks the edge of all four. g-b-r noted that however good the suite, some things "are unlikely to be caught by that, such as data sent to unintended servers or deletions of random files".

We ran the third answer

The most concrete tool from the week was cargo-test-prune, posted as a Show HN on October 25. It runs cargo-mutants, records which deliberate bugs each test catches, and proposes a smaller suite that catches the same ones. The author built it against the pile of tests that accumulates "especially when coding with AI". On the tool's own history, 11 tests caught 73 of 108 mutants; after removing the five it suggested, the remaining six caught "the exact same 73".

We ran its bundled example with cargo-mutants 27.1.0 on commit b3c0fcd. A function positive(value) -> bool has five tests, plus one that is ignored. In 13.8 seconds it generated five mutants, all five were caught, and it proposed removing three tests. One of the three is:

#[test]
fn negative() {
    assert!(!positive(-1));
}

That test checks a real requirement: negative numbers are not positive. It is a removal candidate only because no mutant in this run happened to break negatives without also breaking a test that stays. The tool says so itself. It printed "Confidence: limited to observed mutants. Review specification value before deletion", and the README warns that this result "does not establish that negative-input coverage is dispensable".

The takeaway

Mutation testing answers "does this test detect anything?", and the answer is a number a CI job can enforce. It does not answer "is this the behaviour we wanted?" That question goes back to a person and a specification, which is why the other three camps are not wrong so much as incomplete. A workable split: let the machine prove every test can fail, and spend human review on what the tests are for, the job a generated suite cannot do for itself. We wrote up the generation half in AI test case generation.