A testing agent makes a lot of decisions per run, and almost none of them are interesting.

Which of the 40 elements on this screen is worth touching next. Is this dialog the confirmation step or an unrelated toast. Did the page settle, or is it still loading. Is this error banner the app's fault or the fixture's. Should this recording be kept. A frontier model answers every one of those with a paragraph of reasoning, a tool call, and a bill.

Most of those decisions are a pick from a list you already have. That is the shape Jev fits, TypeSafe AI's System One model: unstructured state in, typed probabilistic decisions out, no prose at any point.

This post is the sorting exercise. Which calls inside a verification or QA agent should become Jev questions, how to shape each one, and which ones must stay in ordinary code no matter how cheap the model gets. It is written from building our own Jev-driven harness and from watching everyone else's launch-week builds. Our harness is not released; it is on an unmerged branch, and there is a waitlist.

What a Jev call actually is

You send a blob of state and a set of typed questions. You get back answers for all of them in one pass. There are three question types, and picking the wrong one is the most common mistake I have made with it.

Choice. One option from a list you supply, up to 255 options. You get a distribution over the options plus a confidence number. This is the workhorse.

Score. A position on a scale you define. Useful when you want to rank or filter rather than decide.

Noul. The probability that a yes-or-no statement is true, as a number from 0 to 1.

Two properties matter more than the type list, and both bite people in week one.

Jev does not see your question ID. If you name a field safe_to_publish and write a vague question, the field name contributes nothing. The model reads the question text and the option descriptions, and that is all. Every requirement has to live in prose that the model actually receives. This is the single most common bug in Jev integrations, because the field name reads like documentation to the human writing it.

Questions in one call run in parallel and cannot read each other's answers. There is no chain of thought across questions in the same call. If question B only makes sense given the answer to A, you need two calls, and you have just paid two round trips.

The three-question test for whether a decision should be a Jev call

Run a decision through these before you write the question.

Can you build the option set in code? If the answer has to be invented (a name, a selector, a free-text summary, a bug title) Jev is out. It cannot emit a string you did not give it.
Is the decision judgement rather than fact? If a deterministic function already knows the answer, use the function. A model asked what the code knows is pure cost with added error.
Is a wrong answer recoverable? Jev returns a probability, not a proof. If the wrong answer is one wasted step, ship it. If the wrong answer is a green check on a broken build, do not.

Point three is the load-bearing one. In our harness, if Jev picks the wrong element from the candidate list, the driver clicks the wrong button, burns a step, and the journal records an action whose declared consequence did not hold. The failure mode is lost coverage, which is measurable and budgetable. That is only true because the model is never asked whether anything worked.

Which decisions in a testing agent are Choice questions

Choice is right when you have an enumerable set of mutually exclusive next moves. In a testing agent that is most of the navigation layer.

What to touch next. Build one option per interactive element off the accessibility snapshot, in code, deterministically:

act:e5      click the button "Create deployment"
fill:f1     fill the textbox "Project name"
finish      Stop driving: the app has been covered.
look_again  Look again without acting, this reading looks incomplete.

The model returns one key. It cannot name an element that is not on the page, because every name in the set was read off the page a moment earlier. A driver that cannot invent a selector cannot hallucinate one. The rule is stated well in a widely read thread on System One models: "never let it invent options. build the candidate list in code, from the DOM, the retriever, the tool trace, then let it pick".

Which consequence this action should have. Instead of asking the model to write an expectation, enumerate the expectation kinds your engine can adjudicate (a request fires, a store slice changes, a route changes, a domain event is emitted) and let it pick one. You get a typed expectation your verifier understands, not a sentence you then have to parse.

Which category of thing just happened. Error banner, permission wall, empty state, loading skeleton, unrelated marketing modal. Five options, one call, no prose.

Add an abstain option, always. Jev cannot decline to answer. Langfuse put it plainly: it is "forced to pick least-wrong answer instead of flagging uncertainty". So uncertainty has to be a member of the option set. Ours is look_again, which costs one snapshot and no action. If your set has no escape hatch, every ambiguous state gets resolved into a confident wrong move.

Keep option descriptions doing the work. Because the question ID is invisible, an option keyed act:e5 means nothing on its own; the description after it is the entire specification.

When to use Score instead of Choice

Score earns its place as a filter in front of a Choice, not as a decision in itself.

A page with 600 interactive elements does not fit a single Choice question usefully, even though the ceiling is 255 options. Retriever AI ran a Score pass first and cut one page from 591 candidates to 82 by relevance to the goal rather than document order. That is the right shape: Score every candidate for goal relevance, take the top slice, then run one Choice over the survivors.

Our own harness caps candidates at 40 in document order, which is the weakest line in the file. Document order is a bad prior and we know it. jev-browser goes to 240 against the DOM.

Score also fits triage: rank the failures in a run by how likely each is to be a real defect rather than a fixture problem, then show the human the top five. You are ordering, not deciding, and an ordering that is slightly wrong still saves the read.

When to use Noul, and the threshold problem

Noul gives you a probability for a yes-or-no claim. It is the tempting one and the dangerous one, because a number between 0 and 1 invites you to pick a threshold and call the result a verdict.

Use Noul for gating cheap work, where both error directions are survivable:

Is this screen worth recording as a flow at all.
Does this state look like the persona's goal has been reached, so the driver should consider stopping.
Is this console error plausibly caused by the action we just drove.

Do not use Noul for "did this work". That is the decision the whole design exists to keep away from a model, and the reason is specific rather than superstitious. Retriever AI's agent asked Jev what to do on a message composer. Jev chose Send at 0.95 probability and 0.94 confidence, and the browser reported no observable effect. Their line is the one to remember: "Neither number establishes that the website accepted the action." Jev answered correctly. Send was the right move. Nobody asked whether Send worked.

On thresholds: the independent jev-ood-calibration study found the probabilities are not calibrated well enough to threshold naively out of distribution. ECE of 0.107 against a 0.024 noise floor, a refit temperature of 2.74, and on a deliberately unknowable task the chosen option carried 0.74 average probability at 44.7% accuracy. Booleans go the other way and come out underconfident at T=0.66, so the sign of the error changes with the question type. Their flat recommendation is to not use the confidence field. If you must threshold, fit the threshold on your own labelled set, per question, and re-fit when the question text changes.

Batch everything into one call, because it is nearly free

Against an identical 17k-character state, we measured seven questions costing 1.029x the input tokens of one. The state dominates; the questions are rounding error.

That changes how you design. Instead of one narrow question per step, ask everything you could plausibly want about this state in a single call: what to do next, what consequence to expect, whether the page looks broken, whether this is a good stopping point, whether an error is ours. Then throw away the answers you did not need. The only constraint is the parallelism rule, so nothing in the batch may depend on anything else in the batch.

The cost floor is low enough that the arithmetic stops being interesting. TypeSafe prices Jev at $0.042 per million input tokens with output free. At roughly 1,000 billed input tokens per decision, 10,000 decisions is about $0.42. For scale on other people's workloads: Hassan El Mghari classified 1,018 AI research papers for $0.08 total at 256ms median latency per paper, and Riley Brown reported 500 emails in seconds for 3.5 cents.

What must stay in code

The list below is not about trust in the model. It is about which decisions have exactly one right answer, and which ones must be answerable without a model in the loop.

Anything with one right answer. In our harness four steps never reach the model at all: start recording, save the flow, finish, and take a snapshot when we have none. Each has a single correct action. Asking a model spends a call to be told what the code already knows, and adds a failure mode that did not exist.

The verdict. Whether an action did what it claimed is decided by a deterministic function reading the recorded evidence, with no model in it. Ours takes eight trust dimensions in and returns one answer. This is the line that makes the cheap driver safe to use: the model steers, and something else decides.

Anything that must produce a string. Naming a saved flow, writing a bug title, summarising a run. Jev cannot emit text. In our comparison run this cost us badly: the frontier driver segmented a long drive into 12 named journeys and the Jev driver kept 1 long recording, because a model that cannot emit a string cannot name anything. If your product is a named regression suite rather than a bug hunt, that gap is the deciding factor today.

The account of what happened. Derive the run report from recorded calls and engine verdicts rather than asking the driver to narrate it. A model summarising its own drive is the one witness with a motive to round "unknown" up to "worked". This applies to frontier models too, and it is a design rule rather than a Jev rule.

Anything where the output goes straight to a human as a reason. Jev returns no rationale. If a person is going to act on "why", something has to write the why, and it will not be this model.

The shape that falls out

Every decision in the loop lands in one of three buckets. Code owns the decisions with one right answer and every verdict. Jev owns the picks from an enumerated set, batched into as few calls as possible, with an abstain option in every set. A larger model owns whatever has to be written in words, which for a driver is mostly naming.

If you are building this, the ordering matters: get the split between driving and deciding right first, then drop the cheap model in. We swapped a frontier driver for Jev on a real payments dashboard and the cost moved by two and a half orders of magnitude while the finding moved by nothing, both arms catching the same 100x under-refund. That result is only available to a design where the model was never the thing deciding. The full run, including the two bugs in our own driver that made the first attempt useless, is in the driver that proved nothing.