Which decisions in your testing agent should be Jev calls

A testing agent makes a lot of decisions per run, and almost none of them are interesting.

Which of the 40 elements on this screen is worth touching next. Is this dialog the confirmation step or an unrelated toast. Did the page settle, or is it still loading.

Is this error banner the app's fault or the fixture's. Should this recording be kept. A frontier model answers every one of those with a paragraph of reasoning, a tool call, and a bill.

Most of those decisions are a pick from a list you already have. That is the shape Jev fits, TypeSafe AI's System One model: unstructured state in, typed probabilistic decisions out, no prose at any point.

This post is the sorting exercise. Which calls inside a verification or QA agent should become Jev questions, how to shape each one, and which ones must stay in ordinary code no matter how cheap the model gets. It is written from building our own Jev-driven harness and from watching everyone else's launch-week builds. Our harness is not released; it is on an unmerged branch, and there is a waitlist.

What a Jev call actually is

You send a blob of state and a set of typed questions. You get back answers for all of them in one pass. There are three question types, and picking the wrong one is the most common mistake I have made with it.

Choice. One option from a list you supply, up to 255 options. You get a distribution over the options plus a confidence number. This is the workhorse.

Score. A position on a scale you define. Useful when you want to rank or filter rather than decide.

Noul. The probability that a yes-or-no statement is true, as a number from 0 to 1.

Two properties matter more than the type list, and both bite people in week one.

Jev does not see your question ID. If you name a field safe_to_publish and write a vague question, the field name contributes nothing. The model reads the question text and the option descriptions, and that is all. Every requirement has to live in prose that the model actually receives. This is the single most common bug in Jev integrations, because the field name reads like documentation to the human writing it.

Questions in one call run in parallel and cannot read each other's answers. There is no chain of thought across questions in the same call. If question B only makes sense given the answer to A, you need two calls, and you have just paid two round trips.

The three-question test for whether a decision should be a Jev call

Run a decision through these before you write the question.

1. Can you build the option set in code? If the answer has to be invented (a name, a selector, a free-text summary, a bug title) Jev is out. It cannot emit a string you did not give it.
2. Is the decision judgement rather than fact? If a deterministic function already knows the answer, use the function. A model asked what the code knows is pure cost with added error.
3. Is a wrong answer recoverable? Jev returns a probability, not a proof. If the wrong answer is one wasted step, ship it. If the wrong answer is a green check on a broken build, do not.

Point three is the load-bearing one. In our harness, if Jev picks the wrong element from the candidate list, the driver clicks the wrong button, burns a step, and the journal records an action whose declared consequence did not hold. The failure mode is lost coverage, which is measurable and budgetable. That is only true because the model is never asked whether anything worked.

Which decisions in a testing agent are Choice questions

Choice is right when you have an enumerable set of mutually exclusive next moves. In a testing agent that is most of the navigation layer.

What to touch next. Build one option per interactive element off the accessibility snapshot, in code, deterministically:

act:e5      click the button "Create deployment"
fill:f1     fill the textbox "Project name"
finish      Stop driving: the app has been covered.
look_again  Look again without acting, this reading looks incomplete.

The model returns one key. It cannot name an element that is not on the page, because every name in the set was read off the page a moment earlier.

A driver that cannot invent a selector cannot hallucinate one. The rule is stated well in a widely read thread on System One models: "never let it invent options. build the candidate list in code, from the DOM, the retriever, the tool trace, then let it pick".

Which consequence this action should have. Instead of asking the model to write an expectation, enumerate the expectation kinds your engine can adjudicate (a request fires, a store slice changes, a route changes, a domain event is emitted) and let it pick one. You get a typed expectation your verifier understands, not a sentence you then have to parse.

Which category of thing just happened. Error banner, permission wall, empty state, loading skeleton, unrelated marketing modal. Five options, one call, no prose.

Add an abstain option, always. Jev cannot decline to answer. Langfuse put it plainly: it "makes Jev pick the least wrong answer instead of saying it does not know".

So uncertainty has to be a member of the option set (more on that and the other structural limits in six things Jev cannot do). Ours is look_again, which costs one snapshot and no action. If your set has no escape hatch, every ambiguous state gets resolved into a confident wrong move.

Keep option descriptions doing the work. Because the question ID is invisible, an option keyed act:e5 means nothing on its own; the description after it is the entire specification.

When to use Score instead of Choice

Score earns its place as a filter in front of a Choice, not as a decision in itself.

A page with 600 interactive elements does not fit a single Choice question usefully, even though the ceiling is 255 options. Run a Score pass first and rank candidates by relevance to the goal rather than document order. That is the right shape: Score every candidate for goal relevance, take the top slice, then run one Choice over the survivors.

Our own harness caps candidates at 40 in document order, which is the weakest line in the file. Document order is a bad prior and we know it. jev-browser goes to 240 against the DOM.

Score also fits triage: rank the failures in a run by how likely each is to be a real defect rather than a fixture problem, then show the human the top five. You are ordering, not deciding, and an ordering that is slightly wrong still saves the read.

When to use Noul, and the threshold problem

Noul gives you a probability for a yes-or-no claim. It is the tempting one and the dangerous one, because a number between 0 and 1 invites you to pick a threshold and call the result a verdict.

Use Noul for gating cheap work, where both error directions are survivable:

• Is this screen worth recording as a flow at all.
• Does this state look like the persona's goal has been reached, so the driver should consider stopping.
• Is this console error plausibly caused by the action we just drove.

Do not use Noul for "did this work". That is the decision the whole design exists to keep away from a model, and the reason is specific rather than superstitious.

Picture a browser agent on a message composer. Jev can choose Send with a probability of 0.95, and the page can still show no observable effect, because the message never left. Neither the probability nor the confidence says the website accepted the action.

Jev answered correctly. Send was the right move. Nobody asked whether Send worked.

On thresholds: the independent jev-ood-calibration study found the probabilities are not calibrated well enough to threshold naively out of distribution. ECE of 0.107 against a 0.024 noise floor, overconfident choice and score answers (refit temperatures of 1.30 and 1.92 after the repo's own correction), and on a deliberately unknowable task the chosen option carried 0.74 average probability at 44.7% accuracy.

Booleans go the other way and come out underconfident at T=0.66, so the sign of the error changes with the question type. Their flat recommendation is to not use the confidence field. If you must threshold, fit the threshold on your own labelled set, per question, and re-fit when the question text changes.

Batch everything into one call, because it is nearly free

Against an identical 17k-character state, we measured seven questions costing 1.029x the input tokens of one (our Jev scorecard). The state dominates; the questions are rounding error.

That changes how you design. Instead of one narrow question per step, ask everything you could plausibly want about this state in a single call: what to do next, what consequence to expect, whether the page looks broken, whether this is a good stopping point, whether an error is ours. Then throw away the answers you did not need. The only constraint is the parallelism rule, so nothing in the batch may depend on anything else in the batch.

The cost floor is low enough that the arithmetic stops being interesting. TypeSafe prices Jev at $0.042 per million input tokens with output free. At roughly 1,000 billed input tokens per decision, 10,000 decisions is about $0.42.

For scale on someone else's workload: Vercel swapped Jev into the safety classifier that checks its agents' commands, and that step got five to 18 times faster.

What must stay in code

The list below is not about trust in the model. It is about which decisions have exactly one right answer, and which ones must be answerable without a model in the loop.

Anything with one right answer. In our harness four steps never reach the model at all: start recording, save the flow, finish, and take a snapshot when we have none. Each has a single correct action. Asking a model spends a call to be told what the code already knows, and adds a failure mode that did not exist.

The verdict. Whether an action did what it claimed is decided by a deterministic function reading the recorded evidence, with no model in it. Ours takes eight trust dimensions in and returns one answer. This is the line that makes the cheap driver safe to use: the model steers, and something else decides. That separation is what AI verification means in practice.

Anything that must produce a string. Naming a saved flow, writing a bug title, summarising a run. Jev cannot emit text. In our comparison run this cost us badly: the frontier driver segmented a long drive into 12 named journeys and the Jev driver kept 1 long recording, because a model that cannot emit a string cannot name anything. If your product is a named regression suite rather than a bug hunt, that gap is the deciding factor today.

The account of what happened. Derive the run report from recorded calls and engine verdicts rather than asking the driver to narrate it. A model summarising its own drive is the one witness with a motive to round "unknown" up to "worked". This applies to frontier models too, and it is a design rule rather than a Jev rule.

Anything where the output goes straight to a human as a reason. Jev returns no rationale. If a person is going to act on "why", something has to write the why, and it will not be this model.

The shape that falls out

Every decision in the loop lands in one of three buckets. Code owns the decisions with one right answer and every verdict. Jev owns the picks from an enumerated set, batched into as few calls as possible, with an abstain option in every set.

A larger model owns whatever has to be written in words, which for a driver is mostly naming. For the same split applied to eleven workloads outside testing, see the Jev playbook.

If you are building this, the ordering matters: get the split between driving and deciding right first, then drop the cheap model in. We swapped a frontier driver for Jev on a real payments dashboard and the cost moved by two and a half orders of magnitude while the finding moved by nothing, both arms catching the same 100x under-refund.

That result is only available to a design where the model was never the thing deciding. The full run, including the two bugs in our own driver that made the first attempt useless, is in the driver that proved nothing.

Fun fact

The 18th-century Mechanical Turk, a chess-playing automaton, had a human chess player hidden inside it. Amazon later named its crowd-work marketplace after it, and Jeff Bezos called the people doing the tasks "artificial artificial intelligence" (IEEE Spectrum). Deciding which calls go to a model and which go to a person is an old trade.

Conclusion

Most decisions in a testing agent are picks from a list you can build in code, and those are the ones worth handing to Jev: what to touch next, which consequence to expect, what just appeared on screen. Anything with one right answer, anything that has to be written in words, and every verdict stay in code or with a larger model.

Take the decisions your agent makes today and run each through the three-question test. Batch the ones that pass into a single call, give every option set a look_again, and keep "did it work" out of the model entirely.

FAQ

Which decisions in a testing agent should be model calls?

The ones that are a pick from an option set you can build in code, that need judgement rather than a known fact, and where a wrong answer costs one step. In a testing agent that is mostly navigation: which element to touch next, which consequence to expect, and what kind of thing just appeared on screen.

What is the difference between Choice, Score and Noul?

Choice picks one option from a list you supply, up to 255, and returns a distribution plus a confidence. Score places each item on a scale you define, which suits ranking and filtering. Noul returns the probability that a yes-or-no statement is true. Choice is the workhorse; Score works best as a filter in front of it.

Should a testing agent ask a model whether a test passed?

No. Whether an action worked should be decided by a deterministic function reading recorded evidence, with no model in it. A probability that an action was right is not evidence the app accepted it: Jev can choose Send at 0.95 probability while the page shows no effect, because the message never left.

Why does every Jev option set need an abstain option?

Because Jev cannot decline to answer. It always returns one of the options you gave it, so an ambiguous screen turns into a confident pick instead of a stall. Add an option like look_again, which costs one snapshot and no action, so uncertainty has somewhere to go.

How many questions can you batch in one Jev call?

As many as you plausibly need, because the state dominates the cost. Seven questions against the same 17k-character state cost 1.029x the input tokens of one. The only rule is that questions in one call run in parallel and cannot see each other's answers, so nothing in a batch can depend on another question.