farol scored a 10x pricing bug as safe to approve. The test that covered it checked nothing

farol launched on Show HN last week. It is an MIT-licensed CLI and GitHub Action that tells the person approving a pull request how risky the merge is: a 0 to 100 score from reach, diff coverage, contract breaks and security findings, with no model and no code execution. The idea is sound and the tool is honest about what it reads. Our view: the score leans on diff coverage, and coverage measures whether a test ran a line, not whether it checked it. So we planted a bug and gave farol three kinds of test.

The setup

A tiny FastAPI service. One endpoint, GET /cart/{cart_id}/total, calls pricing.total(), which applies a coupon. On a feature branch we changed one line, the kind of slip an agent makes in a refactor:

     if coupon == "TEN":
-        return subtotal - subtotal // 10
+        return subtotal - subtotal // 100

The "TEN" coupon now takes 1% off instead of 10%. Every customer using it pays more than they were promised.

We ran farol 0.2.1 (Node 22.23.4) the way its README describes: pytest with coverage first, then farol analyze on the diff with the coverage report, then farol report.

python -m pytest --cov=app --cov-report=json:reports/coverage.json -q
npx -y @winninjs/farol analyze --base main --head HEAD --coverage reports/coverage.json --out farol.json
npx -y @winninjs/farol report --in farol.json

Three tests, three scores

We ran the same diff three times, changing only the test.

The middle row is the one that ships. A test that asserts the total is positive runs the changed line, so farol reports:

### farol · merge risk 17/100 (low) · ◐ quick review
✓ Low risk and nothing to check by hand: likely safe to approve.
- 🟠 `app/pricing.py` **17** · 100% of changed lines covered · 1 of 1 operations
**New code tested:** 100% (1 of 1 changed lines)

"New code tested: 100%." The new code was executed. Nothing tested it. Adding a test that checks nothing cut the risk score from 57 to 17, and that is the core finding.

The third row looks worse than it is. farol does not read test results: the README says it "runs after it (pass or fail)" and "never runs the PR's code". In a real pipeline the failing test would already have turned CI red. The row still shows that the score cannot tell a test that catches the bug from one that does not.

What farol gets right

The reach analysis is real. It found the endpoint as GET /cart/{cart_id}/total and traced it to pricing.py. On a bigger codebase, knowing that one changed file sits under every checkout endpoint is exactly what a reviewer wants first.

It is honest about blind spots. Without lint or audit reports, it prints "not verified" rather than a pass. The README promises it "never calls something safe because it did not understand it", and for what it reads, it keeps that promise. It also states that "farol only informs: it never approves or blocks."

And it uses no model. The author was clear in the thread: "the score is based on static analysis only", from the dependency graph, obvious flaws, and "test coverage of diff lines" (ericholiveira). A deterministic score is the right design for a triage tool.

What it misses

The miss is in the third input. Diff coverage counts executed lines. A line can be executed by a test that asserts on something else, or on nothing useful at all, and coverage will report it as tested. In our run, that one weak assertion moved the verdict from "careful review" to "likely safe to approve".

That is not a farol bug. It is what coverage has always measured. But a merge-risk score turns it into an approval signal, and with agent-written tests the gap gets wider. An agent asked to raise coverage will write exactly the > 0 test above: it runs the line, passes, and checks nothing.

farol does watch tests in one way. Its signals include "Modified existing tests" and "Added suppressions", both "none" in our runs, so an agent that loosens an existing assertion or silences a check would at least be listed for the reviewer. A brand-new weak test is not listed anywhere. It only lowers the score.

Verdict

Use farol for what it is good at: showing where a change reaches, so the riskiest files get read first. Do not read "100% of changed lines covered" as "the change is checked". Before trusting a low score, open the test that covers the risky line and read its assertion. If it would pass with the bug in place, the score is measuring execution, not correctness.

A cheap guard: one assertion on the exact expected value (== 3150) catches this bug, and a mutation-testing run on the changed lines would flag the > 0 test as one that checks nothing.

Caveats. One toy service, one planted bug, farol 0.2.1 with only a coverage report as input; with lint, audit or contract reports the score would include more. farol is a week old and moving fast. The author has already reworked the report since launch.