Hypothesis got faster. Your property tests will not find one more bug until you spend it

Hypothesis 6.170.12, released on 26 October, "reduces Hypothesis' overhead for each test case". The changelog says generating test cases "takes up to a third less time" for simple properties, and shrinking a failure is "up to twice as fast". Our view: take the upgrade, then spend the time it saves. A property test stops after a fixed number of test cases, not a fixed number of seconds. A faster engine runs the same cases sooner and checks nothing new. If the saving shows up only as a shorter CI job, your suite got cheaper and stayed exactly as blind.

Why speed alone changes nothing

Hypothesis's budget is max_examples, and the API reference defines it plainly: "Once this many satisfying test cases have been considered without finding any failing test case, Hypothesis will stop looking." The default profile sets it to 100. A test that runs a third faster still looks at 100 inputs.

There is a second catch. The built-in ci profile, active by default whenever the CI environment variable is set, sets derandomize=True. A derandomized test generates the same inputs on every run. On a laptop, each run at least draws fresh inputs. In CI, an unchanged test checks the same 100 inputs every day, and a faster release makes it check them faster.

The run

We installed hypothesis==6.170.11 and hypothesis==6.170.12 in two fresh venvs on Python 3.13.11, on an Apple-silicon Mac. The code under test is a bill splitter: divide a total in cents among n people and hand out the leftover cents one each. The property is the obvious one: the shares add up to the total.

@given(st.integers(0, 100_000), st.integers(1, 200))
def test_shares_add_up(total, n):
    assert sum(split_bill(total, n)) == total

The planted bug hands out leftover cents only when there are fewer than 190 of them. That needs more than 190 people and an awkward total, so it is rare without being absurd. Every row below used the same 200 seeds on both versions, with the example database off so no run could replay an earlier failure.

We ran nine timing pairs and treated the first three as warm-up. The new version was faster in all nine, and the median saving over the last six was 29%, in line with the changelog's "up to a third".

Three things stand out. First, the speed-up found nothing: 45 seeds caught the bug on both versions. Second, spending the saving did find more. 140 examples on the new version took about as long as 100 on the old one, and caught the bug on 54 seeds instead of 45. Third, under the ci profile the planted bug passed every run at 1,000 examples on both versions and failed only at 5,000. A derandomized test either finds a bug on its first CI run or never finds it, until someone changes the code, the test or the budget.

That last row is not a flaw in the profile. Determinism is why the ci profile exists: a red build should be reproducible, not a dice roll. The cost is that the budget is the only lever left.

What to do with the saving

Keep the release. Then decide where the time goes, on purpose.

• Raise the CI budget. The docs show the pattern directly: settings.register_profile("ci", settings.get_profile("ci"), max_examples=1000). Measure the job before and after. If 6.170.12 bought you 29%, you can buy back about 40% more test cases for the same minutes.
• Keep a randomized lane. Run a nightly job with the default profile and a high max_examples, so the suite explores new inputs somewhere. Hypothesis prints the failing input when it finds one.
• Pin failures as examples. When a nightly run finds something, add it with @example so the deterministic CI lane checks it forever.

None of this is new advice. The settings tutorial covers profiles. What is new is that the upgrade quietly hands every Python team a budget decision, and the default answer is to spend nothing.

The general point

A faster test runner is a gift to the build queue, not to coverage. Coverage moves when something decides to look at more inputs, more states or more paths. Hypothesis made looking cheaper. Whether your suite actually looks harder is a one-line settings change that nobody will make for you.