The Jev playbook: where it wins, where it lags, and the number behind each call

A week in, the question about Jev has stopped being "is it fast" and started being "which of my calls should it take".

This is the answer we arrived at, workload by workload, with the number behind each one. Some of the numbers are ours. Most are not, and every borrowed one is attributed, because the useful thing about launch week is that a lot of people measured a lot of different jobs in public.

The table

The rest of this is the reasoning, and the parts where our own data disagrees with the enthusiasm.

Move it: picking the next action

This is the best fit anyone has found, and several people found it in the same week.

Browser Use put it in an agent that found flights in 7 seconds for $0.0039, with "a new action space every step, DOM state space, small LLM fallback to type". Retriever benchmarked the same shape. We put it in Reticle's verification harness and drove a real payments dashboard.

Our numbers, one run per arm, same app and same loop with only the driver changed:

342x cheaper and the same finding. The reason it fits so well is structural rather than economic: the decision is already a selection from a set the page defines, so a model that can only select is not giving anything up.

Move it: routing, gating, filtering

Three variants of the same shape, all cheap decisions standing in front of expensive ones.

Routing sends the easy request to the cheap model and the hard one to the frontier model. Gating checks a dangerous tool call before it executes. Robomotion ran 6 orders through 55 Jev decisions for $0.0086 and sent anything under 0.55 confidence to a human, which is the pattern in one line.

Filtering is the one people miss. Retriever runs a cheap Score pass that cut an Amazon page from 591 candidate actions to 82 by relevance, then runs Choice over the shortlist. Choice tops out at 255 options anyway, so on a dense page you need this. We do not do it yet, and it is the biggest single improvement available to our driver.

Move it: classification at volume

The clearest wins in launch week were boring high-volume labelling jobs.

Hassan classified 1,018 AI research papers for $0.08 total at 256ms median per paper. Riley Brown put 500 emails through it in seconds for 3.5 cents. Vercel's fx team reported roughly 5 to 18 times faster safety classification than GPT-5.6-luna with better accuracy.

The economics come from one place: Jev prices the state, not the questions. Seven questions cost 1.029x the input tokens of one against an identical state, measured on our own. So ask every question you might want, including the ones you will discard.

Move it carefully: compaction

Scoring every tool call and dropping the irrelevant ones, instead of asking a language model to summarise the transcript, is the highest-leverage trick going. Alex Volkov took a Claude session from roughly a million tokens to 86K in about a second.

The caveat is real though, and it comes from somebody who shipped it: Jev sees the tool calls, not their outputs. So it is scoring the shape of what happened rather than the content, and whether that is good enough is workload-specific. Combined with the 32K window, long sessions have to be chunked.

Depends: rubric grading

Good Start Labs graded 6,003 rubric checks and Jev matched Claude Fable 5.1's verdict 91.5% of the time at $160 per million answers, against $33,000 for Fable itself. DeepSeek V4.1 Flash reached 93.5% for $260.

Those are strong numbers for a grading job, and we wrote a whole post on where they stop being strong. The short version: nobody has broken down the 8.5% disagreement, and it almost certainly concentrates in the hard cases, which are the ones you built the eval to catch. You also lose the rationale, so a wrong grade is silent.

Grade everything with Jev, re-grade the disagreements with a frontier model when you need to know why. That hybrid is cheap and it keeps the explanation where you actually need it.

Keep: anything that must produce a string

This is where our own result argues against the enthusiasm, so it gets the space.

On the same drive that came in 342x cheaper, the Jev arm saved one replayable flow. The frontier arm saved twelve. A saved flow is the whole product of a drive, because it replays forever with no model in it.

The cause is not intelligence, it is vocabulary. A frontier driver names and segments journeys as it goes. A model that cannot emit a string cannot name anything, so ours keeps one long recording and carves it up crudely. Browser Use solved the same problem the obvious way, by keeping a small language model around purely for typing.

If your workload ends in a name, a summary, a coupon code or a JSON body, that part stays with a generating model. Plan for two models, not one.

Keep: open-ended reasoning

On TypeSafe's own aggregate workflow test Jev scores 67.8% against Sol's 74.1%, and on invoice processing 61.8% against 79.1%. It is not a universally better model. It is a much cheaper decision at competitive accuracy on the tasks that are actually selections.

Worth being precise about "cannot hallucinate" too, because it is a claim about types and not about correctness. A review at Kingy put it best: Jev can return department = billing with a perfectly valid probability distribution when the right answer is technical.

Never: deciding whether something worked

Retriever's agent asked Jev what to do on a message composer. It chose Send, at 0.95 probability and 0.94 confidence. The browser reported no observable effect. Their conclusion: neither number establishes that the website accepted the action.

That is not a flaw. It answered the question it was asked, which was what to do next, and Send was the right thing to do next. Nobody asked whether Send worked, and nothing in its answer could have said.

So in our harness the model picks what to try and a deterministic function decides what happened, from evidence, with no model in it. A System One model cannot hallucinate a type, but it still returns a probability, and a probability is not proof.

The thresholds nobody set for you

If you are gating on confidence, read the independent calibration study before you pick a number. Reproducible for about six cents, and the findings that matter:

• ECE 0.107 against a 0.024 noise floor out of distribution, so 4.4x miscalibrated, while on public benchmarks it is nearly perfect.
• Refit temperature 2.74, so choice and score answers are overconfident.
• Booleans go the other way, underconfident at T=0.66. The sign of the error differs by question type, so calibrate per question rather than per model.
• 1,051 of 2,000 option probabilities are exactly 0.00. A 1.00 is often saturation rather than certainty.
• Their flat advice: do not use the confidence field. The max probability was never worse and sometimes much better.

Which is survivable if you place the model correctly, and this is the whole point of the table above. Our completion threshold is 0.7 and I am not going to pretend that is a calibrated belief. It is a tuned knob. If it is wrong, the drive stops early or runs long, and we lose coverage. It cannot produce a false pass, because the model is never asked whether anything passed.

That is the test for any workload on this page. Ask what it costs you when the model is wrong. If the answer is a wasted step, move it. If the answer is a wrong result nobody can see, keep it.

Related

• Which decisions in your testing agent should be Jev calls, for Choice, Score and Noul question design.
• Six things TypeSafe AI's Jev cannot do, for the limits in detail.
• Our measured harness run, including the two bugs in our own driver that made the first version prove nothing.

The harness that produced our numbers is not released. There is a waitlist, and the open-source engine underneath it is free today.