ty, Astral's Python type checker, shipped diagnostic baselines in 0.0.93 on 28 October: record today's errors in a JSON file, and "subsequent checks filter matching diagnostics from CLI output". It is a good adoption tool and it will end up as a merge gate, because "only new errors fail" is exactly the rule teams want on agent pull requests. Our view: use it to adopt ty, not as the gate. The matcher pairs errors by shape, not by place, so a change that fixes one error and introduces a similar one in the same file can pass. We checked.
What the baseline stores
The pull request is unusually candid. It says "This feature is experimental." It explains the choice of matcher: borrowed from basedpyright, it "aligns each file's diagnostics in source order by rule, start and end columns, and number of lines spanned. An error can move down a file and still match."
The file it writes says the same. On our toy project, two invalid-argument-type errors became:
{ "rule": "invalid-argument-type", "startColumn": 21, "endColumn": 24, "lineCount": 1 },
{ "rule": "invalid-argument-type", "startColumn": 35, "endColumn": 38, "lineCount": 1 }
No line number, no message, no hash of the code. That is deliberate: a baseline keyed on line numbers breaks every time someone adds an import. The cost is that two errors with the same rule and the same column span look alike.
The run
We installed ty==0.0.95 (the latest, 29 October) in a fresh venv. One file, app/pricing.py, with a helper cents(amount: int) and two legacy bugs passing strings to it: cents("4") on line 13 and cents("1") on line 18. We ran ty check --update-baseline, then made one edit at a time and ran ty check.
| Edit | Errors in the file | ty check with the baseline |
|---|---|---|
| New wrong call, different columns | 3 | exit 1, shows the new one |
| New wrong call, same columns as a baselined one | 3 | exit 1, shows the new one |
| Functions reordered, nothing new | 2 | exit 0 (correct) |
| A baselined call's wrong string changed to another | 2 | exit 0 |
| Fix line 13, add one new error lower in the file | 2 | exit 1, but shows line 18, an untouched old error |
| Fix both baselined errors, add two new ones in the same order and columns | 2 | exit 0, "All checks passed!" |
| The same two new errors, in a new file | 2 | exit 1, shows both |
The last two rows are the point. In the "fix both" case, the two new errors (cents("5") and cents("2") inside checkout()) are real bugs in code nobody had reviewed, and the gate passed.
The "fix one" row is subtler and in some ways worse. The gate did fail, but the error it printed was on line 18, which the change never touched. A reviewer seeing an old error flagged as new will reasonably assume the matcher hiccuped and run --update-baseline. That writes the actual new error into the baseline, where it stays.
Is this a rare case?
The toy run was built to hit it. Real repositories are the better measure, and the PR has one. Codex checked every tenth commit across 1,000 commits each of Home Assistant and Airflow, and for the chosen matcher the table reads "17 / 158" and "6 / 87": new diagnostics hidden, then old ones shown again. That is 17 of 211 new errors hidden in Home Assistant and 6 of 315 in Airflow. The author also notes, "Sequence alignment exposed 49 of the remaining 72 and hid 23."
So, roughly one new error in twenty-three went unseen over a long history, in projects where the edits were made by people. The PR also says plainly that "I had Codex run this analysis and write this section", which is a nice touch: an agent measured the false-negative rate of a gate that agents will be judged by.
Agent edits push in the wrong direction. An agent asked to "fix the type errors in pricing.py" fixes the old ones and writes new code in the same file, often in the same style, so the new mistakes share the old ones' shape. Fix-one-break-one in a single file is the normal shape of an agent pull request.
What to do instead
Baselines are worth using to adopt ty on a codebase with hundreds of errors. Two guards make them safer as a gate:
- The baseline may only shrink. Fail CI when a pull request adds entries to
ty-baseline.json. A re-baseline then needs a human to say so. - Count, do not just filter, for touched files. For each file the change touched, compare its total error count, run without the baseline, against the base branch. If the count did not drop when baseline entries were removed, something new took their place.
Both are a few lines of shell around git diff --name-only and ty check --output-format concise.
Caveats. One toy file, built to show the case; the hidden rate in practice is the PR's measurement, not ours. The feature is in preview, and the release notes say "the baseline format may change". The same matcher, and the same trade-off, ships in basedpyright.
Sources
Reticleverdict// The agent clicked "Pay". Did the right things actually happen?
Example from the README
Your agent says it works. Reticle checks the network, state and console inside your running app, and names the line to fix.
85 of 88 seeded bugs caught in the public benchmark