Signal
What changed for verifying software
Dated notes on dev tooling, AI coding and testing. Each one runs something, measures something, or cites enough sources to be worth your time.
TeardownReactionDigest
DigestTool-call correctness in agents4 min read
100% valid tool calls is not 100% right ones: HN on constrained decoding in vLLM
Ubicloud took Kimi's tool calls on vLLM from 92% to 100% schema accuracy. HN argued what that number proves. We sorted the thread and did the chain arithmetic.
TeardownWhat diff coverage proves5 min read
farol scored a 10x pricing bug as safe to approve. The test that covered it checked nothing
We ran farol 0.2.1, the new merge-risk tool, on a planted pricing bug. A test that asserts nothing cut the risk from 57 to 17, the same as a test that fails.
DigestVerifying LLM-assisted rewrites4 min read
Rewriting Postgres with LLMs: HN on how you check the copy against the original
Two Hacker News threads on pgrust argued how to verify an LLM-assisted rewrite. We sorted every method they proposed by what it catches and where it breaks.
DigestSoftware quality, then and now4 min read
Software used to be horrible. HN listed which bugs died, and what killed them
A 329-comment HN thread argued whether software got worse. We mapped each bug class it named to the mechanism that ended it, and asked what holds the floor now.
ReactionTest runner caches and stale passes4 min read
pytest 9.2 fixed a test that passed after you broke it. Your app code can still do it
pytest 9.2.0 stops a stale cache passing a test edited to fail. We ran 50 trials per case: test edits are fixed, edits to the code under test still go stale.
TeardownWhat Lighthouse scores prove5 min read
We got a perfect Lighthouse score on a checkout that ships the wrong order
A Lighthouse 100 thread hit the HN front page. We scored 100 in all four categories on a shop that confirms 1 mug when you ask for 3, then ran a full user flow.
ReactionTesting security controls by their effect4 min read
GoAhead loaded the revocation list and never checked it. The test that passes either way is the problem
GoAhead's critical TLS bug loaded a CRL but never checked it. We reproduced the trap in OpenSSL and Python: the setup test passes and the revoked cert gets in.
TeardownSocratic AI debugging tools5 min read
Nietzschian Debugger promises questions, not answers. We said "I don't know" until it solved the bug
We ran the questioning debugger six times on a swallowed 500, saying "I don't know" at every turn we could. All six still found the cause within six questions.
DigestOwning agent-written code4 min read
When agents write the code, HN says review the boundaries, not every line
We sorted all 22 top-level HN comments on owning code that agents write. The most useful answers said where to put the checks, not what you need to know.
ReactionAgent review evals as a merge signal5 min read
Entire's Trails score pushes with review agents. Our Slop eval passed a double discount 6 of 6 times
Entire swaps the pull request for agent eval scores. We ran its default evals on code with five known bugs: the findings caught three, the scores let them pass.
DigestAI-written tests5 min read
Who checks the tests the agent wrote? Four answers from HN
Engineers on HN split four ways on AI-written tests: nobody reads them, read them, make them fail first, or keep the writers apart. We ran the third one.
TeardownLint configs for coding agents5 min read
Teardown: BeautifulRust's lint config went green on five live bugs
We ran BeautifulRust's clippy config on an invoice crate with six planted bugs. It flagged three, then passed a lint-clean copy with five bugs still live.
ReactionAgent decision records4 min read
ADRs for AI asks every agent decision for a falsifier. Make it one you can run
ADRs4AI makes agents state what would prove each decision wrong. We read every falsifier in its public example: 3 of 10 could be checked by a script.
TeardownVision-model assertions4 min read
Teardown: Midscene.js passed a failed save 10 times out of 10
We ran Midscene.js aiAssert 60 times on four bugs. It caught the visible one every time and passed a 500 and a double save every time, with a confident reason.
ReactionCoding benchmarks and their graders4 min read
Kimi K3.1 Pro trails Opus 5.5 by 0.7 points. The grader moves more than that
Kimi K3.1 Pro's Terminal-Bench 4.0 score sits 0.7 points behind Opus 5.5. An audit of the same benchmark's graders moved one model by 26 points. Read the grader.
DigestSoftware quality and defect rates4 min read
Is software getting worse? 46 HN comments, two numbers
We sorted all 46 top-level comments on HN's software-quality debate. Sixteen say worse, seven say better, and two cite a number. The metric nobody tracks.
DigestVulnerability triage5 min read
PHP's CVE week on HN: finding bugs got cheap, proving them did not
We read the HN threads on PHP's two RCE fixes. AI made finding bugs cheap; reproducing reports, deciding fixes and building them in private did not get cheaper.
TeardownCitation-backed AI output5 min read
Teardown: an AI wiki where every fact is cited, checked against the mail
We sampled 30 facts from the Enron Brain's Dynegy page and checked each against its cited email and the original CMU corpus. What a citation proves, and what it does not.
ReactionModel pricing4 min read
LLM prices fell up to 80%. A model-driven test run is cheaper now, and still a sample
Price cuts from OpenAI and Anthropic make re-driving a test suite with a model affordable. We priced it. Cost was never the strongest objection: variance is.
TeardownLighthouse5 min read
Lighthouse gave our broken checkout 100, ten times. Its report listed both charges
We ran Lighthouse 30 times on two live pages and a planted checkout that charges twice. The score measures loading well. It moved 25 points with no code change.
TeardownUI linting5 min read
impeccable detect catches real contrast bugs. It passed a Pay button that does nothing
We ran impeccable's detect CLI, trending on GitHub this week, on five live pages and two planted ones. It is fast and deterministic, and it never clicks.
DigestFormal verification5 min read
Hacker News read the formal verification news and asked one question: proved against what?
We coded all 54 top-level replies in HN's thread on LLM-written proofs. Almost a third were about the spec, not the proof. The same gap sits under AI-written tests.
DigestContext engineering4 min read
HN on context engineering: nobody agrees when to reset, everyone agrees what to keep
We read all 40 comments on Drew Breunig's context engineering post. Reset points ran from 128k to 800k tokens. What survived a reset was a written file.
ReactionC++ contracts4 min read
C++26 failed its ballot over contracts. We compiled one: same code, four outcomes
ISO rejected the C++26 draft, most likely over contracts. We built one postcondition in GCC 16.1 four ways. Two builds exit 0 on a double charge, one silently.
ReactionBrowser automation5 min read
A browser that skips layout should not be checking your UI by default
Moli, a fast headless browser for agents, skips real layout unless asked. We measured what that does to Playwright's visibility and hit-testing answers.
TeardownCode review5 min read
Farol tells a reviewer where to look. It cannot tell them whether it works
We ran Farol, the PR risk-map CLI from today's Show HN, on two real Next.js commits. Its reach analysis was precise and deterministic, with two blind spots.