Every launch-week post about Jev leads with a ratio. Ours did too: a 342x cost reduction on a real payments dashboard, with the same bug found in both arms.

The ratios are real and they are also the least useful thing to know before you build on it. What you need is the list of things the model structurally cannot do, because those are the ones that will not improve with a better prompt, and one of them can put a false green in front of a human.

This is that list, from building a verification harness on it and from reading what other people hit first. Where a limit is someone else's finding it is attributed. Our own Jev harness is unmerged and not released; there is a waitlist if you want it when it lands.

Jev cannot abstain

A System One model returns one of the options you gave it. There is no "I do not know" unless you put one in the set. Langfuse named this first in their evals write-up: it is "forced to pick least-wrong answer instead of flagging uncertainty".

The consequence is that ambiguity in your input does not show up as ambiguity in your output. It shows up as a confident pick. On a screen the model cannot parse, a driver with no escape hatch does not stall, it clicks something.

The fix is boring and mandatory: make abstention an explicit option in every Choice set. In our driver it is look_again, described as "look again without acting, this reading looks incomplete". It costs one snapshot, moves nothing, and is picked often enough to have earned its place. If you have a set of options and none of them means "not enough information", you have a set that lies.

Two follow-on costs that people miss. Your abstain option has to be described as attractively as the real options, because the model reads descriptions and nothing else. And you need a loop guard, since an agent that can always abstain can abstain forever.

Jev gives no rationale, so a wrong answer is silent

There is no reasoning trace. You get an option, a distribution and a confidence number. When a frontier model is wrong, you usually get a paragraph explaining the wrong thing, and the paragraph is the debugging artefact. With Jev, a wrong answer and a right answer look identical from the outside.

This changes how you debug. The only thing you can inspect after the fact is the question text, the option descriptions, and the state blob you sent. All three have to be logged verbatim or the call is not reproducible. Logging the chosen option and moving on will leave you staring at a bad decision with no way to ask why.

It also rules Jev out of any job where the output is a reason a human will act on. Code review comments, bug reports, failure explanations, anything where "because" is the product. Jev can decide which bucket a thing belongs in. It cannot tell anyone why.

Jev does not read your field names

This one catches almost everybody, and it looks like it is working right up until it is not.

Jev does not see your question ID. Naming a field safe_to_publish or is_destructive_action contributes exactly zero instruction. The model receives the question text and the option descriptions. Nothing else. A schema where the identifiers carry the meaning and the prose is thin will behave like a schema with no requirements at all, because that is what it is.

So write the requirement into the question. "Does this text contain a claim about a named individual that is not supported by the source?" is a question. unsupported_claim is a variable name. The same rule applies to options: a key of act:e5 means nothing, and the description beside it is the entire specification of what picking it implies.

Questions in one call cannot see each other's answers

Batching is close to free. Against an identical 17k-character state we measured seven questions costing 1.029x the input tokens of one. That is a real invitation to ask everything at once.

The catch is that questions in a single call run in parallel and cannot read each other's answers. There is no dependency graph and no chain across them. If B is only meaningful given A, you need two calls and two round trips, and at that point the cheap-batching argument no longer applies to that pair.

In practice this pushes you toward flat question sets over decision trees. Anything you would naturally express as "first classify, then ask the follow-up for that class" has to be flattened into questions that all stand alone, or split across calls.

The 32K context window is a cliff, not a slope

Jev holds 32K of context. This is rarely mentioned in the launch coverage and it is the limit most likely to take your integration down in production rather than degrade it.

A large model given too much input gets vaguer. Jev given too much input fails the call. For anything driving a browser, a real application DOM will exceed 32K on ordinary pages, so the state you send has to be a reduction you build and maintain: an accessibility snapshot, a filtered element list, a summary of network activity. That reduction is now part of your system, it has its own bugs, and it is where the information you needed goes missing.

Langfuse also report context rot on longer inputs, so the practical ceiling is below the stated one. Treat 32K as a hard failure boundary and target well under it.

Calibration is worse than the numbers suggest, out of distribution

The confidence field looks like the obvious lever for a threshold. The best available evidence says do not pull it.

The independent study at scienthoon/jev-ood-calibration is reproducible for about six cents and reports, out of distribution:

ECE 0.107 against a 0.024 noise floor.
A refit temperature of 2.74, which is a large correction.
On a deliberately unknowable task, the chosen option carried 0.74 average probability at 44.7% accuracy.
Booleans go the other way and are underconfident at T=0.66, so the sign of the error depends on the question type.
1,051 of 2,000 option probabilities came back exactly 0.00, which makes the distribution much sparser than a probability vector suggests.

Their flat advice is to not use the confidence field. If you are going to threshold anyway, fit the threshold on your own labelled data, per question, and re-fit whenever the question text changes, because the question is the model's entire instruction.

The zeros deserve their own note. If half your option probabilities are exactly zero, "the distribution over options" is not a soft ranking you can use as a relevance score for the also-rans. It is mostly a pick plus a few runners-up.

The one that will cost you a release

Everything above is a limit you can design around. This one is a category error, and it is the reason our verdict has no model in it.

A probability that an action is correct is not evidence that the action worked.

Retriever AI measured this cleanly. Their agent asked Jev what to do on a message composer. Jev chose Send at 0.95 probability and 0.94 confidence. The browser reported no observable effect. Their conclusion: "Neither number establishes that the website accepted the action."

Jev was not wrong. Send was the right move. The question asked what to do next, and it answered what to do next. Nobody asked whether Send worked, and no amount of calibration work closes that gap, because the gap is between two different questions.

This matters because it is exactly the shape of a false green. An agent that asks a model "did that work?" and gets 0.95 back has learned nothing about the application and has acquired a number that looks like it means something. Our verdict is produced by a deterministic function reading recorded evidence across independent channels, and swapping the driver model changed nothing about it. That is not caution about Jev specifically. The same rule kept frontier models out of the verdict for a year before Jev existed.

Where Jev is simply the wrong tool

Short list, stated plainly so nobody has to discover these by shipping.

Anything that produces a string. Names, titles, summaries, messages. In our comparison run the frontier driver segmented a long drive into 12 named journeys and the Jev driver produced 1, because a model that cannot emit a string cannot name anything. A saved, named flow is the durable output of a drive, so for building a regression suite rather than hunting a bug, the expensive driver is currently the right buy.

Open-ended sets. If the answer space is not enumerable in code, there is no question to ask. The 255-option Choice ceiling is generous and it is still a ceiling.

Sequential reasoning inside one call. See the parallelism rule above.

Long state. See 32K.

Anything a deterministic function already knows. Four steps in our harness never reach the model, because each has exactly one right answer. Asking a model to confirm what the code knows buys a new failure mode and nothing else.

What survives all of that

A model that cannot write, cannot abstain, cannot explain and cannot hold much context is still worth building on, for one reason: in the right architecture its errors are cheap and legible.

If Jev picks the wrong option from a set we built in code, the driver clicks the wrong button, burns one step, and the journal records an action whose declared consequence did not hold. The failure is lost coverage. Coverage is measurable, and you can buy more of it at $0.042 per million input tokens with output free, which puts ten thousand decisions at roughly $0.42.

The same wrong answer inside a harness that lets the model generate its own actions and grade its own work is a confidently wrong claim that something worked. Same model, same error rate, completely different blast radius. Which limit hurts you is a property of your design rather than of the model, and that is the part worth getting right before the cheap inference tempts you into skipping it.

The full run, both drivers, same dashboard, including the two bugs in our own driver that made the first attempt useless, is in the driver that proved nothing.