Flaky tests: root causes, not retries

The checkout spec failed on main at 9:40. Someone clicked "re-run", it passed, and the standup moved on.

That is how most teams handle a flaky test: as a reliability nuisance, fixed with a retry. It is a signal problem. The test is reading something that is not the program's truth, so its result is a coin flip, and a retry only flips the coin again.

Every flake lives somewhere in your test harness: in what it waits for, what it shares between tests and what its assertions read. This guide covers the six causes behind most of them, how to find them on purpose, and how to fix each one.

What is a flaky test?

A flaky test is a test that passes and fails on the same code. It happens because the test asserts on something that varies between runs, such as timing, shared state, a network response, an animation, test order or the clock, instead of on what the program actually did. The flakiness is in the test's evidence, not in luck.

Google's definition is the operational one: a result is flaky when the test "exhibits both a passing and a failing result with the same code". Their testing blog reported about 1.5% of all runs flaky, and almost 16% of their tests with some level of flakiness.

So the flaky tests meaning that matters for fixing them is this: the test depends on something the code does not control.

The six root causes

A widely cited study of flaky testing, by Luo, Hariri, Eloussi and Marinov (FSE 2014), examined 201 flakiness fixes from 51 open-source projects and classified 161 of them. Async waiting alone was 45% of them.

In a browser suite, concurrency, the study's second cause, tends to show up in the first and fifth rows in a browser suite: two things racing, or one test's work leaking into the next.

Why retries make it worse

Retries feel like flaky test management. They are a way to stop looking.

Pass rates stop meaning anything. With two retries, a test that fails one run in three still shows green about 96% of the time (1 minus 1/3 cubed). The dashboard stays green. Nobody can tell a healthy test from one that fails a third of the time.

The team learns to ignore red. Google reported that about 84% of the pass-to-fail transitions in their post-submit CI involved a flaky test. When most reds are noise, re-running becomes the reflex, and the reflex does not switch off for the one red that is real.

The real defect ships later. A flake on a timing path is often the only test exercising the slow case. Retry it into green and the bug reaches production, where it surfaces weeks later with no failing run, no diff and nobody who remembers the change. Healing runners do the same to selectors; self-healing tests covers when that repair is safe.

Flaky test detection, on purpose

Waiting for flakes to find you is the expensive way. Four ways to find them first:

1. Repeat on one commit. Same code, many runs. Any disagreement is flake by definition.

Pick the count with arithmetic, not hope. A test that fails 2% of the time shows up at least once in 50 runs only about 64% of the time; 150 runs gets you to 95%. Ten runs proves very little about a rare flake.

npx playwright test checkout.spec.ts --repeat-each=50 --workers=4
go test ./... -count=50 -run TestCheckout
pytest tests/test_checkout.py --count=50   # needs the pytest-repeat plugin

2. Shuffle the order. Order bugs hide behind a stable order.

npx jest --randomize --seed=1234   # checked on Jest 29.7; Jest 28 has no --randomize
go test ./... -shuffle=on          # prints the seed; rerun with -shuffle=<seed>

3. Bisect against a known-good commit. If a test turned flaky recently, git bisect run finds the change. The script must run the test many times, or bisect will chase the coin.

git bisect start HEAD v2.4.0
git bisect run sh -c 'npx playwright test checkout.spec.ts --repeat-each=20'

4. Quarantine with a deadline. Move a confirmed flake out of the blocking suite, keep it running and reporting, and give it an owner and a date. Playwright's --fail-on-flaky-tests flag makes CI fail when a test only passed on retry, which stops new flakes from hiding.

Fix patterns, by cause

Timing. Replace every fixed sleep with a wait for the event the code produces: a response, a store update, a message on a queue. Luo et al. found 54% of async-wait fixes used a wait-for of this kind.

Shared state. Each test builds its own data and leaves nothing behind. If setup is slow, snapshot the database once and restore it per test rather than sharing rows.

Network. Stub third parties at the boundary and assert on the request your code made. A 429 from a shared sandbox is the sandbox's mood, not your bug. Keep one separate, non-blocking contract test that hits the real service, so you still learn when it changes.

Animation. Turn animations off in the test environment, and wait for the state that ends the transition, not the frame where it looks finished.

Order and pollution. Find the polluting test with a fixed shuffle seed, then fix the polluter, not the victim. Luo et al. found 74% of order-dependency fixes cleaned shared state between runs.

Clock and locale. Pass time in, or use your framework's fake timers. A test that adds a month to "today" passes all year and fails on January 31, when JavaScript turns February 31 into March 3. Set TZ and the locale in the test config so a runner in another region produces the same strings.

Assert on state, not pixels

Most of the table comes down to one mistake: the test checks a proxy for the outcome instead of the outcome.

A test that waits for a spinner to disappear measures the spinner. A test that asserts the order exists and the charge returned 200 measures the product. Here is the first kind, in Playwright 1.56.1:

import { test, expect } from "@playwright/test";

test("checkout places the order", async ({ page }) => {
  await page.goto("/checkout");
  await page.getByRole("button", { name: "Pay" }).click();
  await page.waitForTimeout(1000); // "long enough" on the laptop that wrote it
  expect(await page.locator(".spinner").isVisible()).toBe(false);
  await expect(page).toHaveScreenshot("order-placed.png");
});

And the replacement, which waits for the charge and then reads the order back:

import { test, expect } from "@playwright/test";

test("checkout places the order", async ({ page, request }) => {
  await page.goto("/checkout");
  const charge = page.waitForResponse((r) => r.url().endsWith("/api/charge"));
  await page.getByRole("button", { name: "Pay" }).click();

  const res = await charge; // waits for the real response, however long it takes
  expect(res.status()).toBe(200);
  const { orderId } = await res.json();

  const order = await request.get(`/api/orders/${orderId}`);
  expect(order.ok()).toBe(true);
  expect((await order.json()).status).toBe("paid");
});

We ran both 20 times on the same commit, against a small fixture shop whose payment API answers in 200 to 1,400 ms. The shop and both tests are in a public repo: npm install && npm run demo reruns it. The latency is random, so your failure count will differ from ours.

$ npx playwright test tests/before.spec.ts --repeat-each=20 --workers=1
  ✓   1 tests/before.spec.ts:3:5 › checkout places the order (1.2s)
  ✘   2 tests/before.spec.ts:3:5 › checkout places the order (1.1s)
  ✓   3 tests/before.spec.ts:3:5 › checkout places the order (1.2s)
  ✘   4 tests/before.spec.ts:3:5 › checkout places the order (1.1s)
  ...
    Error: expect(received).toBe(expected) // Object.is equality
    Expected: false
    Received: true
  8 failed
  12 passed (30.9s)

$ npx playwright test tests/after.spec.ts --repeat-each=20 --workers=1
  20 passed (25.2s)

The first test is right about the app and wrong about time: whenever the charge took longer than a second, it failed a working checkout. The second test cannot fail that way, because it waits for the event instead of guessing how long it takes, and it would also catch a charge that returned 500 behind a cheerful toast.

The same idea scales past one test. When an agent writes and checks the code, the verdict should come from the requests, the app state and the console, not from a model's reading of the page. That is the argument in AI verification, and it is how flows found by exploratory testing become regression checks that do not flake.

Reticle replays saved flows as assertions on network, state and console, with no model in the loop: eight replays, one verdict, 0% flake, and about 480 tokens per four-flow re-run, roughly 251 times cheaper than an LLM re-drive. The open-source SDK install takes a few minutes.

When to delete a test

Sometimes the design is the problem. A test that needs five sleeps to check a dialog is telling you the dialog has no observable "done" state. Add one, an event or a status field, and the test gets simple.

Delete the test when it duplicates a deterministic lower-level test, when nobody owns the feature, or when stabilising it means asserting nothing. A deleted test costs coverage you can see. A flaky one costs trust in every other test. Which layer a check belongs in to begin with is covered in our guide to automated testing.

A 30-day flake-reduction plan

Week 1: measure. Record every test that passed only on retry, and run the suite 20 times on one commit overnight. Rank by failure rate.

Week 2: quarantine. Move every flake out of the blocking suite, each with an owner, a suspected cause from the table and a date.

Week 3: fix by cause. Timing first. Sleeps become event waits, screenshots become state checks, and the unfixable get deleted.

Week 4: lock the door. Fail any test that needed a retry, and run shuffle and repeat nightly so new flakes surface in a day, not a quarter.

Fun fact

A bug that disappears when you look at it is called a Heisenbug, and the name is often credited to the computer scientist Jim Gray, who used it in a 1985 paper on why computers stop.

He did not coin it. The earliest known use is a 1983 ACM debugging symposium, whose proceedings say only that such a bug "was called a Heisenbug by one participant". Whoever named the unreproducible bug is, fittingly, unreproducible.

Conclusion

A flaky test asserts on something the program does not control: a timer, shared state, a third party, an animation, test order or the clock. Retries hide that and teach the team to ignore red.

Find flakes by repeating and shuffling, fix them by cause, and assert on what the system did. A test that reads the program's truth has no coin to flip.

FAQ

What is a flaky test?

A flaky test passes and fails on the same code. It depends on something that varies between runs, such as timing, shared state, network responses, animations, test order or the clock, instead of only on what the program did. The variation is in the test's evidence, not the code.

What causes flaky tests?

The most common cause is waiting on asynchronous work with a fixed delay: a 2014 study of 161 flaky-test fixes found 45% were async waits. Other causes are concurrency, test order dependency, shared state between tests, third-party network responses, animations and reliance on the real clock, timezone or locale.

Do retries fix flaky tests?

No, retries hide them. A test that fails one run in three still shows green 96% of the time with two retries, so pass rates stop meaning anything and the team learns to re-run red builds. The cause stays, and the slow path it exercised goes untested.

How do you detect flaky tests?

Run the same test many times on one commit and look for any disagreement, for example with Playwright's --repeat-each or go test -count. Shuffle test order to expose dependencies, record every test that passed only on retry, and bisect with repeated runs to find the commit that caused it.