Jev vs LLM-as-judge: what $160 per million graded answers actually buys

LLM-as-judge is the default way to grade anything an AI produces. A rubric goes into a prompt, a frontier model reads the answer, and out comes a score with a paragraph of justification. It works, and at volume the bill is the reason most eval suites run on a sample instead of everything.

Jev, TypeSafe AI's System One model, changes that arithmetic hard enough that "grade every row" becomes a normal thing to do. It also removes several things you probably rely on without noticing.

I build a verification tool, so treat the enthusiasm accordingly. The most useful number in this post is an 8.5% disagreement rate that somebody else measured, and the most useful section is the one about what a judge cannot tell you at any price.

What the cost difference actually is

The concrete datapoint comes from Good Start Labs, who ran 6,003 rubric checks through both (reported by Langfuse). Jev matched Claude Fable 5.1's verdict 91.5% of the time, at $160 per million graded answers against $33,000.

That is roughly 200x, and the shape of it matters more than the multiple. Jev is priced at $0.042 per million input tokens with output free.

Output being free is most of the story for grading, because a judge's output is a score plus a justification, and the justification is usually the larger half of the bill. Remove the prose and the prose disappears from the invoice.

Latency moves too. Vercel swapped Jev into the safety classifier that checks its agents' commands, and that step got five to 18 times faster.

What that buys you is not a cheaper eval run. It is a different sampling policy. If grading the full set costs less than a coffee, you stop arguing about which 500 rows to grade, and you stop discovering that the interesting failures were outside the sample.

What that 91.5% is hiding

Agreement of 91.5% is either excellent or unusable, and which one depends entirely on what lives in the other 8.5%.

If the disagreements are spread evenly across easy and hard cases, you have a slightly noisier judge and the volume more than pays for it. If they concentrate on the ambiguous cases, you have a judge that agrees with the expensive one on everything you did not need help with.

Nobody has published that breakdown, mine included, so this is the thing to measure on your own rubric before you switch anything.

Two facts make me expect concentration rather than spread. First, Jev cannot abstain. Langfuse put it directly: it "makes Jev pick the least wrong answer instead of saying it does not know".

A frontier judge given a genuinely ambiguous answer will often hedge in its prose, and a human reading the eval picks that up. Jev returns a clean pick with a number attached.

Second, the calibration evidence says the number does not rescue you. The independent jev-ood-calibration study found that on a deliberately unknowable task the chosen option carried 0.74 average probability at 44.7% accuracy out of distribution, with ECE of 0.107 against a 0.024 noise floor. Choice and score answers needed a refit temperature above 1 (1.30 and 1.92 after the repo's own correction), which means overconfident.

Booleans went the other way and came out underconfident at T=0.66. Their recommendation is to not use the confidence field at all.

Put those together: on the cases where you most want the judge to tell you it is unsure, it will hand you a confident-looking pick and a probability you have been advised not to trust. The practical answer is to build the uncertainty into your rubric as an explicit option, which is the next section.

How rubric design changes when the judge cannot write

Moving a rubric from a prompt to a typed question set is not a port. Several things you were doing implicitly stop existing.

The requirement has to be in the question text. Jev does not see your question ID. A field named factually_supported contributes no instruction whatsoever; the model reads the question and the option descriptions and nothing else. Rubrics written for frontier judges often lean on the field name plus a one-line hint, because the big model fills the gaps. This one does not fill gaps.

Each grade needs a written-out option. With Choice you supply up to 255 options and get back a distribution plus a confidence value. A five-point rubric becomes five options, each with a description that says what qualifies. The descriptions are where all the rubric detail goes that used to live in a paragraph of preamble.

"Cannot tell" has to be an option. Otherwise ambiguity becomes a grade.

Score exists and is often the better fit. For a rubric that is genuinely a scale rather than a set of categories, Score gives a position on a scale you define. Noul gives the probability that a yes-or-no statement is true, which suits binary checks like "does this answer cite a source it was not given".

Grade the whole rubric in one call. Against an identical 17k-character state we measured seven questions costing 1.029x the input tokens of one (our Jev scorecard). The graded artefact dominates the token count, so a fifteen-criterion rubric costs almost exactly what a one-criterion rubric costs.

The constraint is that questions in a call run in parallel and cannot read each other's answers, so a rubric with conditional criteria ("if it scored 3 or above, also check X") has to be flattened or split across two calls.

The graded thing has to fit in 32K. That is the room for state plus the longest question (64K per request in total). Long transcripts, large documents and full DOM snapshots do not fit, so a reduction step becomes part of your eval pipeline, with its own failure modes. The full list of limits is in six things Jev cannot do.

The rationale you are giving up

A frontier judge writes down why. That paragraph is doing more work in most eval suites than people admit.

It is how you debug a rubric. When a judge starts marking good answers as failures, you read ten justifications and find that the criterion is ambiguous. With Jev you have the pick, the distribution and nothing else, so rubric debugging becomes a labelling exercise against a human set rather than a read-through.

It is also how disagreements get adjudicated. "Model said 2 out of 5" is not something you can argue with in review. "Model said 2 out of 5 because the answer never addresses the second half of the question" is.

The workable pattern is a hybrid, and it is the one I would recommend to anyone with an existing judge-based suite: grade everything with Jev, then re-grade the disagreements and the low-margin cases with a frontier model for rationale.

At a 200x cost gap you can send 5% of your volume to the expensive judge and still land near the cheap price. You get full coverage plus prose exactly where a human is going to read it.

When LLM-as-judge is still the right call

Small volumes. If your eval set is 200 rows that run nightly, the frontier bill is already noise and the rationale is worth more than the savings.

Rubrics that are still changing. Early rubric iteration is mostly reading justifications and noticing that the criterion says something other than what you meant. Do that with a model that talks.

Open-ended criteria. If the grade is not enumerable, there is no Choice to offer.

Anything a human will act on directly. Content moderation appeals, code review, grading that a student or a customer sees. The reason is the product.

For the same call made workload by workload, with a number behind each, see the Jev playbook.

What no judge can tell you, at any price

The last section is the one I actually care about, because it is the mistake I see most often in agent evaluation and it does not depend on which model you use.

A judge grades an artefact. It does not observe a system.

Picture a browser agent on a message composer. Jev can choose Send with a probability of 0.95, and the page can still show no observable effect, because the message never left. Neither the probability nor the confidence says the website accepted the action.

Jev answered correctly. Send was the right move. The question was what to do next, and it got an answer to what to do next. Whether Send worked is a different question entirely, and no confidence value on the first one is evidence about the second.

Swap in a frontier judge and nothing changes. Ask GPT or Claude "did this action succeed?" over a DOM snapshot and you get a well-argued paragraph that is grounded in a screenshot of a page that renders from the same state the handler set. A page agreeing with itself proves nothing.

We saw exactly this on a real payments dashboard (the full run): a refund dialog showing the correct rupee amount while the POST body underneath carried a value one hundredth of the true one, twice, with the repository's own Playwright suite passing 7 of 7 over it. Every surface a judge could read was telling the truth.

So the split we hold to is: a model may decide what to do, and a deterministic function decides whether it worked, reading recorded evidence from a channel other than the one that performed the action. Ours takes eight trust dimensions in and returns one answer, with no model in it.

When we swapped a frontier driver for Jev on that dashboard, the cost moved by two and a half orders of magnitude, the token volume fell about 50x, and the finding moved by nothing. Both arms caught the money bug, because neither arm was the thing deciding.

That separation is the whole idea behind AI verification, and the result is available to anyone whose architecture separates steering from judging. It is not available to an agent that grades its own work, whatever it costs per million tokens.

The full run, the two bugs in our own driver that made the first attempt worthless, and the column where the cheap driver loses badly, are in the driver that proved nothing. Our Jev harness itself is on an unmerged branch rather than released; the waitlist is here.

Fun fact

The MT-Bench authors tried a "repetitive list" attack on LLM judges: take an answer with a numbered list, have GPT-4 reword the list without adding anything, and put the padded copy next to the original. Claude-v1 and GPT-3.5 preferred the longer, identical-in-substance answer 91.3% of the time; GPT-4 fell for it 8.7% of the time (Zheng et al., 2023, Table 3). Judges, like the rest of us, can be impressed by word count.

Conclusion

Jev turns LLM-as-judge from a sampled expense into something you can run on every row: $160 per million graded answers against $33,000, with 91.5% agreement in one published comparison. What you give up is the written rationale, the option to abstain and anything over 32K.

Before switching, measure where your own 8.5% lives. Run both judges on one rubric, read the disagreements, and keep the frontier judge for the cases that land there.

FAQ

What is LLM-as-judge?

LLM-as-judge is grading AI output with another model: a rubric goes into a prompt, a model reads the answer, and it returns a score with a written justification. It is the default way to run evals on open-ended output, and at volume its cost is why most suites grade a sample instead of every row.

Is Jev a good replacement for an LLM-as-judge?

Jev is a good replacement for an LLM-as-judge on high-volume, enumerable rubrics. In Good Start Labs' 6,003 rubric checks it matched a frontier judge 91.5% of the time at about 200x lower cost. It cannot write a rationale or abstain, so keep a frontier judge for disagreements, changing rubrics and open-ended criteria.

How much does LLM-as-judge cost compared to Jev?

In Good Start Labs' comparison, a frontier LLM-as-judge cost $33,000 per million graded answers and Jev cost $160. Jev is priced at $0.042 per million input tokens with output free, which matters for grading because a frontier judge's written justification is usually the larger part of the bill.

Can an LLM judge tell whether an agent's action worked?

No, an LLM judge cannot reliably tell whether an agent's action worked, because it grades an artefact rather than observing the system. A page can show the right state while the request underneath was wrong. Whether an action worked needs recorded evidence from a channel other than the one that performed it.