LLM-as-judge is the default way to grade anything an AI produces. A rubric goes into a prompt, a frontier model reads the answer, and out comes a score with a paragraph of justification. It works, and at volume the bill is the reason most eval suites run on a sample instead of everything.

Jev, TypeSafe AI's System One model, changes that arithmetic hard enough that "grade every row" becomes a normal thing to do. It also removes several things you probably rely on without noticing.

I build a verification tool, so treat the enthusiasm accordingly. The most useful number in this post is an 8.5% disagreement rate that somebody else measured, and the most useful section is the one about what a judge cannot tell you at any price.

What the cost difference actually is

The concrete datapoint comes from Good Start Labs, who ran 6,003 rubric checks through both. Jev matched Claude Fable 5.1's verdict 91.5% of the time, at $160 per million graded answers against $33,000.

That is roughly 200x, and the shape of it matters more than the multiple. Jev is priced at $0.042 per million input tokens with output free. Output being free is most of the story for grading, because a judge's output is a score plus a justification, and the justification is usually the larger half of the bill. Remove the prose and the prose disappears from the invoice.

Latency moves too. Vercel's fx team reported roughly 5x to 18x faster safety classification than GPT-5.6-luna with improved accuracy on their task. Hassan El Mghari classified 1,018 AI research papers for $0.08 total at 256ms median latency per paper.

What that unlocks is not a cheaper eval run. It is a different sampling policy. If grading the full set costs less than a coffee, you stop arguing about which 500 rows to grade, and you stop discovering that the interesting failures were outside the sample.

What that 91.5% is hiding

Agreement of 91.5% is either excellent or unusable, and which one depends entirely on what lives in the other 8.5%.

If the disagreements are spread evenly across easy and hard cases, you have a slightly noisier judge and the volume more than pays for it. If they concentrate on the ambiguous cases, you have a judge that agrees with the expensive one on everything you did not need help with. Nobody has published that breakdown, mine included, so this is the thing to measure on your own rubric before you switch anything.

Two facts make me expect concentration rather than spread. First, Jev cannot abstain. Langfuse put it directly: the model is "forced to pick least-wrong answer instead of flagging uncertainty". A frontier judge given a genuinely ambiguous answer will often hedge in its prose, and a human reading the eval picks that up. Jev returns a clean pick with a number attached.

Second, the calibration evidence says the number does not rescue you. The independent jev-ood-calibration study found that on a deliberately unknowable task the chosen option carried 0.74 average probability at 44.7% accuracy out of distribution, with ECE of 0.107 against a 0.024 noise floor and a refit temperature of 2.74. Booleans went the other way and came out underconfident at T=0.66. Their recommendation is to not use the confidence field at all.

Put those together: on the cases where you most want the judge to tell you it is unsure, it will hand you a confident-looking pick and a probability you have been advised not to trust. The practical answer is to build the uncertainty into your rubric as an explicit option, which is the next section.

How rubric design changes when the judge cannot write

Moving a rubric from a prompt to a typed question set is not a port. Several things you were doing implicitly stop existing.

The requirement has to be in the question text. Jev does not see your question ID. A field named factually_supported contributes no instruction whatsoever; the model reads the question and the option descriptions and nothing else. Rubrics written for frontier judges often lean on the field name plus a one-line hint, because the big model fills the gaps. This one does not fill gaps.

Each grade needs a written-out option. With Choice you supply up to 255 options and get back a distribution plus a confidence value. A five-point rubric becomes five options, each with a description that says what qualifies. The descriptions are where all the rubric detail goes that used to live in a paragraph of preamble.

"Cannot tell" has to be an option. Otherwise ambiguity becomes a grade.

Score exists and is often the better fit. For a rubric that is genuinely a scale rather than a set of categories, Score gives a position on a scale you define. Noul gives the probability that a yes-or-no statement is true, which suits binary checks like "does this answer cite a source it was not given".

Grade the whole rubric in one call. Against an identical 17k-character state we measured seven questions costing 1.029x the input tokens of one. The graded artefact dominates the token count, so a fifteen-criterion rubric costs almost exactly what a one-criterion rubric costs. The constraint is that questions in a call run in parallel and cannot read each other's answers, so a rubric with conditional criteria ("if it scored 3 or above, also check X") has to be flattened or split across two calls.

The graded thing has to fit in 32K. That is the whole context window. Long transcripts, large documents and full DOM snapshots do not fit, so a reduction step becomes part of your eval pipeline, with its own failure modes.

The rationale you are giving up

A frontier judge writes down why. That paragraph is doing more work in most eval suites than people admit.

It is how you debug a rubric. When a judge starts marking good answers as failures, you read ten justifications and find that the criterion is ambiguous. With Jev you have the pick, the distribution and nothing else, so rubric debugging becomes a labelling exercise against a human set rather than a read-through.

It is also how disagreements get adjudicated. "Model said 2 out of 5" is not something you can argue with in review. "Model said 2 out of 5 because the answer never addresses the second half of the question" is.

The workable pattern is a hybrid, and it is the one I would recommend to anyone with an existing judge-based suite: grade everything with Jev, then re-grade the disagreements and the low-margin cases with a frontier model for rationale. At a 200x cost gap you can send 5% of your volume to the expensive judge and still land near the cheap price. You get full coverage plus prose exactly where a human is going to read it.

When LLM-as-judge is still the right call

Small volumes. If your eval set is 200 rows that run nightly, the frontier bill is already noise and the rationale is worth more than the savings.

Rubrics that are still changing. Early rubric iteration is mostly reading justifications and noticing that the criterion says something other than what you meant. Do that with a model that talks.

Open-ended criteria. If the grade is not enumerable, there is no Choice to offer.

Anything a human will act on directly. Content moderation appeals, code review, grading that a student or a customer sees. The reason is the product.

What no judge can tell you, at any price

The last section is the one I actually care about, because it is the mistake I see most often in agent evaluation and it does not depend on which model you use.

A judge grades an artefact. It does not observe a system.

Retriever AI measured this on a browser agent. Their agent asked Jev what to do on a message composer. Jev chose Send at 0.95 probability and 0.94 confidence, and the browser reported no observable effect. Their conclusion is the sentence to keep: "Neither number establishes that the website accepted the action."

Jev answered correctly. Send was the right move. The question was what to do next, and it got an answer to what to do next. Whether Send worked is a different question entirely, and no confidence value on the first one is evidence about the second.

Swap in a frontier judge and nothing changes. Ask GPT or Claude "did this action succeed?" over a DOM snapshot and you get a well-argued paragraph that is grounded in a screenshot of a page that renders from the same state the handler set. A page agreeing with itself proves nothing. We saw exactly this on a real payments dashboard: a refund dialog showing the correct rupee amount while the POST body underneath carried a value one hundredth of the true one, twice, with the repository's own Playwright suite passing 7 of 7 over it. Every surface a judge could read was telling the truth.

So the split we hold to is: a model may decide what to do, and a deterministic function decides whether it worked, reading recorded evidence from a channel other than the one that performed the action. Ours takes eight trust dimensions in and returns one answer, with no model in it. When we swapped a frontier driver for Jev on that dashboard, the cost moved by two and a half orders of magnitude, the token volume fell about 50x, and the finding moved by nothing. Both arms caught the money bug, because neither arm was the thing deciding.

That result is available to anyone whose architecture separates steering from judging. It is not available to an agent that grades its own work, whatever it costs per million tokens.

The full run, the two bugs in our own driver that made the first attempt worthless, and the column where the cheap driver loses badly, are in the driver that proved nothing. Our Jev harness itself is on an unmerged branch rather than released; the waitlist is here.