pytest 9.2 fixed a test that passed after you broke it. Your app code can still do it

pytest 9.2.0 shipped on 27 October, and its most important fix is one line in the changelog: a test file edited twice in the same second, at the same size, could run its old version. So a test you had just broken could still pass. Our view: upgrade today, but do not read the fix as "pytest now always runs what is on disk". We measured it, and the same stale pass still happens when the second edit lands in the code under test instead of the test.

That gap matters more now than it did when people typed every edit. Tools that apply several edits in a row can land two writes inside one second.

What changed

Issue #13292 describes the mechanism. pytest rewrites your test modules so assert can print useful diffs, and caches the result in __pycache__. The release notes say "the pyc header could only record whole-second timestamps", so a file rewritten within the same second, with the same byte count, looked unchanged. "The cache is now keyed on a hash of the source."

Same size is not exotic. Change 1 == 1 to 1 == 2, or return 1 to return 2, and the byte count does not move.

We ran it 50 times

The script writes a passing test, runs pytest, rewrites the test to fail at the same size, and runs pytest again. No sleeps, no touch, no timestamp tricks: just two writes as fast as a script makes them, on Python 3.13.11 on macOS, a fresh directory per trial.

printf 'def test_x():\n    assert 1 == 1\n' > test_a.py
pytest -q -p no:cacheprovider test_a.py      # passes, as it should
printf 'def test_x():\n    assert 1 == 2\n' > test_a.py
pytest -q -p no:cacheprovider test_a.py      # should fail

Case	pytest	Second run passed a failing test
Test file edited	9.1.1	44 of 50
Test file edited	9.2.0	0 of 50

The fix does what it says. On 9.1.1, the runner reported green for a test that asserts 1 == 2 in 44 of 50 trials.

The part the fix does not cover

Then we kept the test fixed and edited the module it imports, which is what usually changes when an agent fixes a bug:

printf 'def f():\n    return 1\n' > calc.py     # test asserts f() == 1
pytest -q -p no:cacheprovider test_calc.py   # passes
printf 'def f():\n    return 2\n' > calc.py     # same size, now wrong
pytest -q -p no:cacheprovider test_calc.py   # should fail

Case	pytest	Second run passed a failing test
Module under test edited	9.1.1	29 of 30
Module under test edited	9.2.0	27 of 30
Module under test edited, `PYTHONDONTWRITEBYTECODE=1`	9.2.0	0 of 30

The reason is upstream of pytest. Your app modules are imported by CPython's normal machinery, not by pytest's rewriter. PEP 552 describes that pyc format as carrying "the source timestamp, and the source file size", and says "the current timestamp invalidation method will remain the default". pytest fixed the cache it owns. The interpreter's cache still works on the same one-second clock.

We also tried --assert=plain on 9.1.1, which skips rewriting: 28 of 30 stale passes, because the test file then goes through the same timestamp check.

The other quiet fixes

The release has a theme: the runner reporting something that was not true.

#14445: assertion rewriting evaluated walrus expressions more than once, "which reported results the test never computed".
#14635: parametrized fixtures in a conftest.py "could be silently dropped depending on the order of the specified paths".
#14638: top-level options in pytest.toml were "silently ignored" without a [pytest] table. Now they error.

Each one is a green or a config that meant less than it looked. None would show up as a failure, which is exactly why they lasted.

What to do

Upgrade to 9.2.0 (PyPI). It is a minor release, and the fixes above are the reason.
Where an agent or a script edits and re-runs in a tight loop, set PYTHONDONTWRITEBYTECODE=1 for the test run (docs). In our trials it took both cases to zero. The cost is recompiling on every run, which a small suite will not notice.
Treat a test run as evidence about the code that was loaded, not about the code on disk. If the two can disagree for a second, a pass after a fast edit deserves one more run.

Caveats

How often a real agent hits the one-second window, we did not measure. A model call between edits usually takes longer than a second; batch edits, codemods and formatters do not. Our runs are a worst case by design: back-to-back writes on a local disk. Filesystem timestamp precision varies, and other platforms may give different counts.