What smoke testing is, and what should run on every commit

What is smoke testing?

Smoke testing is a small, fast set of checks run on every new build to decide whether it is worth testing further. It covers only the few paths the product cannot live without, such as signing in and paying. If any of them fails, the build is rejected at once and no deeper testing starts.

That is the whole idea of smoke testing in software testing: a gate, not an inspection. It answers one question, "is this build broken in an obvious way?", and it answers it in minutes. It is also called a build verification test or a confidence test.

A smoke suite is one of the things a test harness runs, usually the first, and the fastest layer of automated testing. The definition is the easy part. The hard part is choosing what goes in, and keeping it small once an agent is writing most of the commits.

Where the term comes from

The phrase comes from electronic hardware testing: you power on a new board, and if smoke comes out, you turn off the power and stop (Kaner, Bach and Pettichord, Lessons Learned in Software Testing, p. 95). Plumbers used a similar check, pushing smoke through a closed pipe system to find leaks before running water through it. Software kept the logic: a cheap first check that saves you from wasting an expensive one.

Smoke testing vs sanity testing vs regression testing vs build verification

These terms get used loosely, and some sources treat smoke and sanity testing as synonyms. The working distinction most teams use is this:

So smoke testing vs sanity testing is breadth against focus: smoke is wide and shallow across the whole product, sanity is narrow and deep on one change. Smoke testing vs regression testing is speed against coverage: smoke runs on every commit and checks a handful of paths, regression runs less often and checks everything. Build verification is often folded into smoke, and the table separates it because "the server starts" and "a user can pay" are different questions.

The time budget: speed is the entire point

A smoke suite exists to give an answer before the person who made the change has moved on. Continuous integration has had a number for this for more than two decades: the Extreme Programming guideline of a ten-minute build, which Martin Fowler calls "perfectly within reason" for most projects (Continuous Integration). The smoke stage is a slice of that build, so it has to be a few minutes at most.

Speed matters more than coverage here because of what people do with a slow gate. If it takes long enough to start the next task, the result arrives after attention has moved on. People batch commits to avoid waiting, so a red result now points at five changes instead of one. Then someone makes the stage non-blocking "for now", and it stops being a gate.

To get under the budget, cut in this order:

1. Anything that is not a critical path. Settings pages, profile edits and admin screens belong in regression.
2. Duplicate coverage. Ten tests that all start with "log in" are one login check and nine regression tests.
3. Real waits. A test that sleeps for an animation or a debounce is paying for time it does not need. Wait on the request or the state change.
4. Environment setup per test. Seed once, share read-only fixtures, and reset only what a test writes.

A smoke suite that takes forty minutes is not a smoke suite. It is a regression suite running at the wrong time, and it delays every commit to say what a smaller check would have said sooner. What belongs in the regression run depends on what each tool can see, which is how we sort regression testing tools.

What belongs in a commit-time smoke suite when an agent writes the commits

This is where most guides stop, and where the job has changed. When a coding agent writes a large share of the commits, there are more of them, they arrive faster, and the author reports success in confident prose whether or not the change works. The smoke suite is now the first independent check on work nobody watched being done.

That changes what belongs in it.

The three or four journeys that carry revenue or access. For most products that is signing in, the core action the product exists for, paying, and whatever grants or removes access. If any of these breaks, nothing else about the build matters. Pick them by asking "what would we roll back for at 3 a.m.?", not "what is easy to automate?".

Outcomes, not appearances. Each journey asserts what the user actually needs to be true:

• Sign-in: the session exists and the next request is authorised, not just that the dashboard heading rendered.
• Checkout: one charge request fired and returned success, and the order is in a paid state. Not "a confirmation appeared".
• The core action: the record was written and reads back with the value entered.
• Access: a user without the permission gets refused by the server, not merely shown a hidden button.

Why "the page rendered" is not a smoke test. A page can render from a cache, from mock data left switched on, or from an error boundary with a friendly message. An agent's change can break the request behind a page while leaving the page looking perfect. A smoke check that asserts only on what is visible will pass in exactly the cases where the agent's own summary would also have said "done". It adds no information.

What to leave out. Visual polish, edge cases, rarely used settings and anything an agent explores better than a script. Those belong in regression runs and in exploratory testing, where breadth is the point.

Wiring it to every commit

Make the smoke stage the first job after the build, and make everything else wait for it.

# .github/workflows/ci.yml (excerpt): smoke runs first and blocks the rest
jobs:
  smoke:
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run build
      # playwright.config.ts starts the app via its webServer option
      - run: npx playwright test --grep @smoke --max-failures=1
  regression:
    needs: smoke
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npx playwright test --grep-invert @smoke

Where it runs: on every push to a branch with a pull request open, and on main. For agent-written branches, run it on every push, since an agent may push many times before asking for review.

What it blocks: the merge, and every later CI job. timeout-minutes keeps the budget honest: a smoke stage that cannot finish in five minutes fails, which forces the conversation about what to cut. --max-failures=1 stops at the first red, because the build is already rejected.

What it reports: which journey failed, the step, and the evidence: the failed request with its status, or the state that was wrong. "Smoke failed" with a link to a log is not a report. Post it on the pull request, where the agent or the person can act on it.

If an agent drives the journeys, record each one once and replay the recording on every commit with no model in the loop, so the smoke stage gives the same verdict for the same commit.

How a smoke suite grows into a forty-minute suite

Steve McConnell, writing about daily builds in 1996, expected the smoke test to evolve: it might start in seconds and grow to "30 minutes, an hour, or more" as the system grew (Daily Build and Smoke Test). That was for a daily build. On every commit, the same growth breaks it. It happens four ways:

1. Every incident adds a test. A bug escapes, a check is added to smoke "so it never happens again", and nobody moves it to regression later.
2. Coverage becomes the goal. Someone reports smoke coverage as a metric, and the suite grows toward it.
3. Retries hide flaky tests. A smoke test that fails sometimes gets a retry, and each retry multiplies the stage's worst-case time.
4. Setup grows with the product. Each new service adds start-up time, and the suite starts waiting for things none of its journeys touch.

The fix for all four is a rule that adding to smoke requires removing something, or a time budget that fails the stage when it is exceeded.

Grade your smoke suite: seven checks

1. Does it finish within the time budget you set, on every run?
2. Does it cover only the journeys you would roll back for?
3. Does each check assert an outcome (a request, a state) rather than only what is visible?
4. Does a failure block the merge, with no "allow failure" flag?
5. Does it give the same verdict on the same commit every time?
6. Does a failure name the journey, the step and the evidence?
7. Did anything get removed the last time something was added?

Fun fact

Teams used to make breaking the build memorable. McConnell's 1996 article describes groups that handed a lollipop to each "sucker" who broke the build, to be taped to their office door until it was fixed, and others where the culprit wore goat horns or paid $5 into a morale fund (McConnell, IEEE Software, 1996). An agent cannot wear goat horns, which is one more reason the smoke check has to be strict.

Conclusion

Smoke testing is a fast gate: a handful of critical journeys, run on every commit, that reject a broken build before anything slower starts. Keep it to the paths you would roll back for, assert outcomes rather than appearances, and hold it to a time budget. When an agent writes the commits, that small strict suite is the first check that does not take the author's word for it.

FAQ

What is smoke testing?

Smoke testing is a small, fast set of checks run on every new build to decide whether it is worth testing further. It covers the critical paths, such as signing in and paying. If any check fails, the build is rejected immediately and deeper testing does not start. It is also called build verification or confidence testing.

What is the difference between smoke testing and sanity testing?

Smoke testing is broad and shallow: it checks that every critical path works at all, on every build. Sanity testing is narrow and focused: it checks that one specific change or fix works, usually on a build that already passed smoke. Some teams use the terms interchangeably, so agree on the definitions inside your team.

What is the difference between smoke testing and regression testing?

Smoke testing checks a handful of critical paths in minutes, on every commit, and rejects a broken build. Regression testing checks that everything that used to work still works, takes much longer, and usually runs before a merge, a release or nightly. Smoke is the gate; regression is the inspection that follows it.

How long should a smoke test take?

A few minutes. The commonly cited continuous integration guideline is a ten-minute build, and smoke is only one stage of that build, so it should take a small share of it. If your smoke suite takes much longer, it has grown into a regression suite and should be trimmed back to critical paths.

Is smoke testing manual or automated?

It can be either, but on every commit it has to be automated. Manual smoke checks still make sense before a release or after a deployment, when a person clicks through the critical paths in production. A smoke test in software that runs per commit must be fast, scripted, and able to block the merge.