Back to Blog

Replacing LLM judgment calls with Jev

Stop paying writing prices for answers that are just numbers

AI Native

When you're adding AI to your system, it's easy to hook up an LLM, use structured output to get it to respond in JSON, then validate that and use it in your program.

It works great, and it's so flexible, so why not? The thing is, these LLMs are so flexible and smart that it's like driving in a nail with a sledgehammer. It's complete overkill.

But until recently we didn't have general-purpose models that were flexible and easy enough to do something else. Now we do.

In Zenable, we ask models a lot of questions, for instance:

  • Does this requirement apply to this file?
  • How risky is this change?
  • Is this proposed requirement good enough to show a person?
  • Which Policy as Code engines can express this rule?

And dozens more. Every one of those has a small, known set of answers. Asking a generative model to read a prompt, reason in prose, format a JSON object, and hand back a single label means you're paying writing prices, in both latency and cost, for an answer that's just a number. Then we coerce that JSON into a typed object and run more validation on it (with Pydantic, for example) before our code can use the answer.

Typed judgment instead of text generation

Instead of forcing a generative model to act like a classifier, you can use a model built to answer typed questions about a piece of state and return probabilities. It doesn't write replies or code. Better yet, you can bundle several questions about the same state into one request and run them in parallel.

We implemented this using TypeSafe's Jev, and it covers three kinds of question. Sometimes you just need a yes or no answer (what Jev calls a Noul), like whether a requirement applies to a file. Sometimes you've got a list of options and you need the probability of each one (a Choice), like whether a verifier engine can express a requirement fully, partly, or not at all. And sometimes you're grading against a rubric (a Score), like how risky a pull request is.

Narrowing in two steps

Once judgment is fast and cheap, it changes how you structure the rest of your pipeline.

In Zenable, requirements already attach to scopes, so a rule only applies to the repositories, paths, and file types it was written for. That first step is deterministic and essentially free, but on a large codebase it can still leave a lot of rules in play for a small change.

Typed judgment adds a second narrowing step. Before a generative model ever sees the diff, Jev judges which of the remaining scoped rules actually bear on the code in front of it. Only those rules get passed along. If models are given too many instructions, they start to forget some of them, and their output quality drops because of the distractions. So keeping the set of requirements focused on only the ones that are relevant and in scope significantly improves finding quality and reduces false positives. The numbers below are strictly cost and latency.

Putting it into practice

To see what actually changed, we ran both paths on the same frozen inputs through production code (real merged pull requests and a real requirement set). And this is how that changed our runtime:

WhereWhat it decidesLatencyCost
PR risk assessmentHow risky each changed file is74% faster70% cheaper
Zenable CLI (check)Which requirements apply to this fileAbout the same54% cheaper
Context refinementWhich proposed context edits to deliver51% faster16% cheaper
Verifier engine selectionWhich engines can express a requirement~6x faster~98% cheaper
Proposed-requirement gradingWhich proposals passAbout the sameAbout the same

Surprisingly, zenable check did not get faster. Zenable checks run both deterministic verifiers and AI reviews on each file. Jev didn't affect the deterministic runtime, but for the AI reviews we used Jev to pick which requirements apply to the file before a generative model reviews it. That cut the cost by more than half, but the review was already a single fast call that took about two seconds, and the Jev step adds about a second in front of it. The smaller prompt made the generative call about twice as fast on most files, which only pays that second back, and three of our five test files took longer with Jev than without.

But for most of the scenarios that we tested, we saw a dramatic speedup and cost reduction, with the same (in some cases, even better) levels of quality.

PR risk assessment saw the biggest drop because it now makes several small typed calls in parallel where it used to make one large generative call. Verifier engine selection was the most dramatic, roughly 6x faster and 98% cheaper, because we stopped using a generative model to pick from a short, fixed list.

The real payoff: knowing when a call was close

Cutting latency by 74% and cost by 70% is great, but the bigger engineering win is getting calibrated probabilities back instead of bare labels. As you know, you can ask an LLM to give you a probability or an explanation, but those should be considered qualitative at best. Decision models are calibrated, so their probabilities are a core part of what training optimizes (the step that measures model quality).

Ask a generative model how risky a change is, and it will tell you "medium" whether the call was obvious or a total coin flip. Ask it how sure it is, and it will give you a number, but nothing ties that number to how often it's right.

A calibrated probability tells you which one it was. When a score lands near a threshold, your code can route that borderline PR to a human instead of guessing. We also rank results and take the top of the list, because answers on the cut line can shift between identical requests.

Calibration isn't automatic, though. TypeSafe recommends checking calibration on your own data, and we set our thresholds against our own inputs rather than trusting defaults.

What we'd tell another team

If you're building AI-native workflows and still using generative models to grade, route, or classify, three things stood out from our rollout:

Don't grade with the same model that wrote the work. Separating the writer from the grader changed what we actually ship more than any cost or latency number did.

Benchmark against your real production path. If your current generative path already uses prompt caching, measure your savings against that, not against uncached list prices.

Look at where your probabilities land before picking a threshold. Run your own data first so you know where the clear calls sit and where the coin flips are.

All of this is live in Zenable today. Generative models are great at writing, reasoning, and synthesizing. Save them for that, and stop paying writing prices for answers that are just numbers.

And we're actively on the lookout for more scenarios where we can make many small, parallel decisions quickly to improve our decision making, our use of LLMs, and even our deterministic verifier generation (more details on that coming soon).