AI test case generation, without the slop

Ask a model for test cases and you get forty of them in ten seconds. They read well, they cover the form, and most of them would never find a bug. That is the quiet problem with AI test case generation: it optimises for cases that look plausible, while the risk sits in the cases nobody thought to write.

This post is about fixing the input. It is one layer of AI test automation, the one that decides what gets tested at all.

What is AI test case generation?

AI test case generation is the use of a language model or another learning system to propose test cases (inputs, steps and expected results) from a source such as a prompt, a specification, recorded user flows or past failures. The source decides the quality. A model can only propose risks that its input already contains.

Automatic test case generation is older than language models. Fuzzers, model-based testing and combinatorial tools have generated cases from grammars and state machines for decades. What changed with test case generation using generative AI is that the input became plain English, and plain English is where most of the risk information gets lost.

What AI test case generation tools actually produce

Run any prompt-driven generator on a checkout form and sort the output. Almost every case falls into one of four piles.

1. Happy-path restatements. "User enters a valid card and completes checkout." It is the requirement, rephrased as a test.

It looks like coverage because it maps one-to-one onto the user story. It finds nothing the developer did not already click through.

2. Permutations of the same path. Valid card with a Visa, with a Mastercard, with an Amex. Valid address in Ohio, in Texas. Each one is a new row in the test plan and the same code path in the app.

They look like coverage because the count goes up and each row is technically distinct.

3. Spec-derived edge cases. Empty field, 256 characters, an emoji in the name, a negative quantity. These are useful, and generators are good at them, because input boundaries are written down in the spec or implied by the field type.

They look like coverage because they are the cases a reviewer expects to see.

4. Duplicates wearing different names. "Verify the order total updates" and "Check the cart sum recalculates" are one test. Generators produce these freely, especially across several prompts.

They look like coverage because the titles differ.

Only the third pile tests something new, and it tests the part of the system the spec already describes. The bugs that reach production usually live somewhere else: in what happens after a retry, between two tabs, or when a later step reads data an earlier step wrote wrong.

The coverage illusion

Case count is easy to measure, so it becomes the target. It is a poor proxy.

In what was then the largest study of its kind, Inozemtseva and Holmes generated 31,000 test suites for five Java systems and found "a low to moderate correlation between coverage and effectiveness when the number of test cases in the suite is controlled for" (ICSE 2014). More tests and more coverage did not reliably mean more bugs caught.

A worked example. A team asks a generator for coupon-field cases and gets 30: valid code, expired code, wrong case, leading spaces, 1 character, 100 characters, and so on. Now list the coupon risks that would cost money:

• the coupon applies twice when the Pay request is retried
• the discount also comes off shipping
• a refund uses the pre-discount total
• the coupon survives after the cart drops below the minimum spend
• two tabs apply the same single-use code

Thirty cases, and none of them touch those five. The suite went from zero to thirty and its risk coverage stayed where it was. Nobody prompted for those five, because nobody had them in mind, which is the whole reason they are risks.

Generate from recorded flows, not prompts

A prompt contains what someone remembered to type. A recorded flow contains what the system did. That difference is the entire fix.

Here is an illustrative recording of one checkout, trimmed, of the kind any browser session capture or HAR file can give you:

{
  "flow": "checkout with coupon",
  "steps": [
    { "action": "click 'Apply'", "request": "POST /api/cart/coupon", "body": { "code": "FALL20" }, "status": 200,
      "state": { "cart.discount": 20, "cart.total": 80 } },
    { "action": "click 'Remove item'", "request": "DELETE /api/cart/items/3", "status": 200,
      "state": { "cart.subtotal": 35, "cart.discount": 20, "cart.total": 15 } },
    { "action": "click 'Pay'", "request": "POST /api/charge", "body": { "amount": 15 }, "status": 200 }
  ]
}

Read it and the cases write themselves, and they are not the cases a prompt gives you:

• The real sequence. Remove an item after applying a coupon. Does the discount survive below the minimum spend? (Here it did: subtotal 35, discount still 20.)
• The real payloads. The charge body carries amount: 15. Replay the Pay step twice and check that one charge exists, not two.
• The real state transitions. cart.discount never changed after the delete. Generate the case that asserts it should.

A model given this recording proposes cases about this app's actual behaviour. A model given "test the coupon field" proposes cases about coupon fields in general. Recorded flows come from real use, from your own sessions or from exploratory testing, where a person or an agent pokes at the app with no script.

Failure history is the second input worth feeding in. Every bug that reached production is a case your generator missed once. Give the model your last 20 incident tickets alongside the recording, and ask for cases in the same family on the flows that changed.

Prioritise by blast radius, not by count

A generated suite of 200 cases does not need 200 equal runs. Rank each case by what breaks if it fails:

Two questions place a case: does a failure touch money, data or access, and how many users hit this path? A spec-derived case on a rarely used settings field is rank 4, however clever it is. A boring replay of the Pay step is rank 1.

Deduplication and maintenance: why generated suites rot

Generated suites rot faster than hand-written ones because nobody owns any single case. Three signals tell you it has started:

1. Cases that never fail. A case that has passed on every run for three months, through real regressions elsewhere, is probably asserting nothing that matters. Delete it or rewrite it.
2. Cases that fail together. If eight cases always go red at the same time, they are one case with eight names. Keep the clearest one.
3. Cases nobody can explain. If no one on the team can say what product rule a case protects, it has lost its reason to exist.

Deduplicate by behaviour, not by title. Two cases that send the same requests and assert the same state change are the same case, whatever they are called.

Where humans still decide

A generator proposes. Three calls stay with a person, because they need knowledge that is not in any recording:

• Intent. The recording shows the discount survived below the minimum spend. Is that a bug, or a deliberate grace rule from marketing? Only someone who knows the product can say.
• Business rules. "Single-use codes" is a rule in a contract or a campaign brief. A model cannot infer it from traffic.
• Whether a case should exist at all. Every case costs run time and attention. Someone has to say no to the hundredth permutation.

This is the split described in AI QA: the model widens the search, and a person decides what counts as correct.

A one-hour exercise for your own feature

Pick one feature that changed recently. Then:

1. 10 minutes: ask your usual generator for test cases from a one-line prompt. Save the list.
2. 15 minutes: use the feature by hand with the network tab open. Export the HAR file, or note each request, payload and status.
3. 10 minutes: pull the last five bugs filed against that feature.
4. 15 minutes: give the model the recording and the bug list, and ask for cases that would have caught those bugs plus anything in the recording that looks wrong.
5. 10 minutes: rank both lists by blast radius and compare the top five of each.

In most teams the second list's top five look nothing like the first's. That gap is what your prompt-generated suite has been missing.

Automated test case generation tools, by source

Compare automated test case generation tools by what they generate from, not by brand:

The strongest setups combine the last two: record flows, then check every generated case against the bugs you have already had.

Reticle works from that end. It records flows inside the running app, with network, state and console for each step, and replays them as checks with no model in the loop. In its published benchmark, a four-flow re-run costs about 480 tokens, and eight replays of one flow gave the same verdict every time: 0% flake (scorecard). Generated cases still need a person to decide intent.

Fun fact

Test case generation has had a benchmark since 1979. Glenford Myers opened The Art of Software Testing with a self-assessment: write test cases for a program that reads three integers and says whether they form a scalene, isosceles or equilateral triangle. He listed 14 cases, and reported that "highly-qualified professional programmers think of, on the average, only 7-8 of the possible 14" (Myers, chapter 1).

Forty years on, a model will happily produce all 14. The cases nobody thinks of have just moved to the parts of your app that are not a triangle.

Conclusion

AI test case generation is only as good as its input. From a prompt, it produces restatements, permutations, boundary cases and duplicates, and the count hides how little risk they cover. Feed it recorded flows and failure history, rank what it proposes by blast radius, prune cases that never fail, and keep intent and business rules with a person.

FAQ

What is AI test case generation?

It is using a language model to propose test cases (inputs, steps and expected results) from a source such as a prompt, a specification, recorded user flows or past bugs. The source matters most: a model can only suggest risks its input contains, so a one-line prompt mostly yields happy paths and input boundaries.

Can AI generate test cases from requirements?

Yes, and it does it well for what the requirement states: the main path and the input boundaries. It cannot see what the requirement leaves out, such as retries, two tabs or a later step reading bad data. Pair requirement-based cases with cases generated from recorded flows and from your past incidents.

Are AI-generated test cases reliable?

They are reliable as proposals and unreliable as a finished suite. Expect duplicates, permutations of one path and expected results the model guessed. Review each case for intent, deduplicate by behaviour rather than title, rank by blast radius, and delete cases that have never failed through real regressions.

How do you choose AI test case generation tools?

Ask what each one generates from. Prompt-based tools are fast but blind to sequence and state, and spec-based tools cover contracts. Tools that work from recorded flows or failure history find the risks nobody typed. Most teams need one of the last two, and our AI testing tools list shows the field.