Rewriting Postgres with LLMs: HN on how you check the copy against the original

Dan Luu's "We should be more ambitious with the software we build" reached 903 points on Hacker News in late October. His main example is pgrust, a Rust rewrite of Postgres that commenters describe as largely AI-generated, and his case for taking it seriously is its testing: it has been "uncovering real bugs in Postgres". A day earlier, a ClickBench thread argued its benchmark entry relied on per-query kernels and result caching. Our read of both threads: when an LLM rewrites software, the original program is the best test you have, and a fixed benchmark is the worst, because whatever is scored is what gets optimized.

We set the benchmark dispute aside. What is useful is how the commenters proposed checking the copy, because almost every method they named is a verification method with a known failure mode.

The methods, sorted

The oracle is the original

baq put the whole argument in one line: "Rewrite X in Y is an ideal RL task given you have X as an oracle trivially." Postgres is a free, exhaustive answer key. Run the same query on both and compare. No spec, no expected values written by hand, no model judging the result.

That is also why the rewrite found bugs in Postgres itself. When the two disagree, either side can be wrong, and Luu lists several that turned out to be on the original's side.

The same move works at the scale of one function. Before an agent refactors it, keep the old version, run both on the same inputs, and diff the results. The old code is the spec nobody wrote down, and it costs nothing to consult until the day you delete it.

Proof is a tool, not the goal

malisper, pgrust's author, described compiling C and Rust to Lean and having "an LLM write a proof that the C and Rust code behave identically", covering "about ~70k lines of C code". sealeck asked the obvious question: "What if there are bugs in this C,Rust->Lean compiler?"

malisper's answer is the most useful framing in the thread: "My goal isn't to "prove" the software is correct. My goal is to find as many bugs as I can with the least effort." A proof here is a bug-finding method with its own trusted base, judged on what it finds.

More copies, different stacks

visarga suggested N-version programming, with replicas made "maximally different": "choose a different language, framework, agent harness and LLM." TacticalCoder went further: "Just re-writing in Rust isn't that useful: we all move to pgrust and then what?" Their worry is the day the original is retired and the copy has nothing left to disagree with.

The point holds beyond databases. A differential check only works while two independent versions exist. Retire one and you are back to trusting a single implementation, now one that nobody wrote line by line.

Fixed inputs invite fitting

The ClickBench dispute is the same lesson from the other side. bmacho: "My opinion: (some) bench results should run on randomized data." malisper said that before the 0.4-beta release, "we generated a large number of fresh queries and made sure the engine runs them and performs well across all of them".

verdverm noted the project had added "a lint test to make sure it doesn't cheat again", and asked how anyone ensures the agent does not alter that file. A guard the optimizer can edit is a suggestion.

The takeaway

Every method that held up in these threads has the same property: the check comes from outside the thing being checked. An original program, an independent implementation, an input the code never saw. The methods that worried people came from inside: a lint rule in the same repo, a benchmark the code could be shaped around.

That is a useful rule for any agent-written change, not just rewrites of databases. If the agent can see the check, or edit it, assume the check is being fitted. Keep at least one check it cannot reach: an old version to diff against, inputs generated after the change, or a test file outside the agent's write access.

Caveats. Two threads, read by us, limited to comments that proposed a method. We take no position on the ClickBench entry itself; the discussion is in pull request #2165.