Teardown: an AI wiki where every fact is cited, checked against the mail

A Show HN last week, "We had AI write a wiki from 45k Enron emails, every fact cited", describes a model that read 45,047 emails and wrote 214 pages. Every one of their 49,401 facts links to the message it came from. We took one page apart and checked its citations against the original mail. Our view: the citation is the cheap half of the job. What makes this project believable is the part most AI products skip, a separate judge and a published error rate, and that is the bar anything claiming "verified" should meet, a coding agent's "tests pass" included.

What the Enron Brain claims

The front page makes two claims: "93% of sampled facts fully supported by their email" and "0 of 500 sampled facts wrong at their core". The methodology page explains how both were measured, and it is unusually candid.

There are two separate checks. The first is mechanical: every fact quotes a sentence, and the build looks for that sentence in the message. Of 62,205 candidate facts, 37 failed and were dropped. The authors say what this does not show: "A quote that matches proves the sentence exists. It doesn't prove the statement says only what the sentence says."

The second check covers that gap. A different model, Claude Fable 5.1, reads 500 random facts beside their messages and grades each one. After the second verification pass, 466 of 500 were fully supported, 34 added a detail the mail does not state, and none was unsupported at its core.

What we ran

We picked the Dynegy page, the rival that agreed to buy Enron in November 2001 and then walked away. It loads its data from two JSON files: 989 facts, citing 1,551 quoted excerpts from 537 distinct messages. We downloaded the CMU Enron corpus the site was built from and ran two checks of our own.

The quote check is the strongest result, and it matches the site's own. On this page, every published quote is real.

For the 30-fact sample, we read each fact against the full text of every message it cites. Most are close paraphrases of a news article an Enron employee forwarded, such as "Dynegy expects the acquisition to close by the end of the third quarter of 2002." Several merge sources. One fact gives the deal as about $9 billion in stock, roughly $25 billion with $13 billion of debt and $2 billion of preferred stock. The fact cites ten messages, and each figure is in at least one of them.

The closest call was a fact saying lawsuits "accuse Dynegy and the other energy companies" of withholding power. The cited Dow Jones item says "the lawsuits claim Enron and the other energy companies" did it, and two sentences earlier says a similar suit was filed against Dynegy. The fact is fair, but only by joining those two sentences. A detail we expected to be invented, a credit filter "shutting Enron out again", turned out to be in the forwarding note: "Dynegy is shutting us down on ICE again."

Thirty facts cannot tell 93% from 100%. If the true rate is 93%, a perfect run of 30 still turns up about one time in nine. Our sample agrees with their number. It does not improve on it.

What the citations miss

Whether the mail was right. A citation proves the email said it, and much of this corpus is forwarded press coverage. One of the founders said so in the thread: "people make mistakes in communications all the time" (jreynar).

Anything built on top of the facts. Each page splits the relationship into phases, and the methodology calls phases the weakest part: 25 of 540 failed in the third audit. They were revised after that audit, so the re-check "isn't a held-out measurement". The authors say this themselves, which is the right call.

Anything a pattern does not recognise. The site masks phone numbers and "the value after a password, passcode, PIN, username, user ID" label, and warns that "a number or credential written in a form the patterns don't recognize could remain." We found one. A 2001 welcome email for Dynegy's trading platform lists an ID and a password on lines labelled with a trailing hyphen, and both are published unmasked in the page's data file. They are 25 years old and the corpus is public. It still shows that a pattern check passes whatever the pattern never described.

What to take from it

grey-area put the business problem plainly: "one incorrect fact could be very damaging and it's very hard to check them all" (comment). This project's answer is the right one: do not claim to have checked them all. Cite the evidence so anyone can check one. Have a second model grade a held-out sample, publish the rate, and say which numbers were tuned after the fact.

Most AI tools that say "verified" do the first step at most. A coding agent that reports "all tests pass" is citing evidence, and often does not show it. Ask for three things: the evidence it cites, a check by something other than the model that did the work, and a measured rate of how often that check disagrees. The Enron Brain shows all three can be built.