Back to Blog

Only Review the Risky Stuff

Moving from peer-reviewed to risk-assessed code

ChangelogPlusProEnterprise

Coding agents are amazing. They can write code faster than we ever imagined. But we still need to make sure the changes we make align with our requirements, don't have side effects, and work the way we expect. So today, most teams do peer review using pull or merge requests.

But it doesn't take long to find bottlenecks in your most popular repositories. The code owners in those repositories get burnt out from all the code review they're doing, and we realize that this just won't scale.

And on top of that, it's easy for PRs to get bigger, which makes reviews shallower. That complexity makes it easier to bury one genuinely risky change inside 2,000 lines of otherwise pristine code.

We're not the only ones seeing this. Vercel found that over half of the PRs in its largest monorepo were approved with zero comments:

"Requiring a human to approve every CSS tweak and docs update doesn't make the codebase safer, it makes engineers slower and delays basic updates unnecessarily."

— Vercel Engineering, 58% of PRs in our largest monorepo merge without human review

Mandatory peer review was never the actual goal. Managing risk was the goal. For decades, "a second pair of eyes on every commit" was the best proxy we had. Now that coding agents generate code faster than any team can meaningfully read it, that proxy is breaking down.

Writing Code Isn't the Slow Part Anymore

Checking it is. And a lot of engineering and security leaders feel stuck between two bad options. Keep human review on every PR, and the queue keeps growing until approvals become a formality. Or drop human review and let an LLM approve PRs, and now you're trusting a model that can skip a requirement or drift as your codebase changes.

If you've ever had to explain a change to an auditor, a CISO, or a post-incident review, you know "an AI looked at it and said it was fine" isn't going to cut it.

The solution is to make sure each PR gets a mix of deterministic checks and AI review against your requirements, plus a risk assessment. Low-risk PRs can merge once those checks pass, and your (human) reviewers spend their time focusing on the higher-risk changes. Instead of peer-reviewed by default, your code is risk-assessed by default.

How a Risk-Assessed Fast Lane Works

If low-risk changes are going to skip a human, you need to be explicit about how risk is measured and checked before code hits main. Here's how we do it.

It starts with a deterministic baseline: your existing CI tests, linters, and Policy as Code checks, like Semgrep verifier rules compiled from your engineering and security standards. These checks are fast, don't use any inference tokens, apply every rule every time, and leave a record of exactly which controls ran.

But static analysis only catches known patterns. It can't tell you whether the architecture makes sense, whether a product tradeoff is right, or whether a change fits its context. So pair it with a context-aware AI review against your scoped company requirements, plus transparent risk scoring on every PR. Imagine that every PR gets scored across six configurable dimensions, and combined with a set of pre-configured rules like "PRs that touch CODEOWNERS files are always at least high risk", or "PRs that touch sensitive code in repos that process cardholder data are always at least high risk". Then, that resulting risk level (low, medium, high, or critical) is used to inform the depth and type of verifiers (including human review) that are required.

Those rules are deterministic verifiers that act as guardrails, because an LLM shouldn't have the final say on whether a sensitive change skips a human. They can only raise a PR's risk level, never lower it. Zenable ships defaults for CODEOWNERS files and CI/CD pipelines, and you can add your own, like any PR over 1,000 lines, or anything that touches auth, billing, or migrations. You can also tune the rubric per repo or service, so your payments stack and your docs site don't have to meet the same bar. And because those rules don't depend on the model, a change written to game the AI score still gets caught.

Finally, the author stays in control of timing. Low-risk PRs don't merge on their own. Once a PR is assessed low, CI is green, and the AI review passes, it merges only when the author asks, using GitHub auto-merge, a label, or a merge queue. And if you want a second pair of eyes on a low-risk change anyway, add the needs-review label and it goes to a person.

Putting It Into Practice

We run Zenable on Zenable, so our own monorepo went first. Here's how our PRs compare, 12 weeks before risk-assessed merging against the 12 weeks after:

MetricChange
PRs merged per week44% increase
Lines merged per week44% increase
Median PR size22% smaller
Median time from open to merge53% faster
Merged with no human approval38% of all merged PRs (and growing)

Things changed once small, low-risk changes had a way through. We found ourselves naturally breaking work into more, smaller changes that were easier to assess, and easier to revert if we needed to. And because they've become so easy to assess, we haven't had to revert a single low-risk change yet. That's the same idea DevOps has always been built on: small changes are easier to verify, which makes them lower risk, which means they can ship faster. Now it works with coding agents too.

The Real Payoff: Code Review That Keeps Improving

Shipping faster is nice, but for us the bigger win is the feedback loop it builds into code review.

Most teams try to guide coding agents with a CLAUDE.md or AGENTS.md file in the repo. But within a few weeks the codebase moves on, exceptions pile up, false positives get ignored, and nobody has time to keep the rules current. A fast lane is only safe if the checks underneath it keep up with your codebase.

So we treat our review system like production code, and we test it with evals.

When a risk assessment doesn't match your team's judgment, you, or your coding agent, can tell Zenable what was wrong and why. That feedback becomes eval cases, and Zenable uses them to propose specific adjustments that fix the case without breaking your other cases.

You also decide how those evals and rules change. With a human-in-the-loop policy, nothing gets added to your baseline, whether it's a new eval case or a new good or bad example, without a person approving it. Or, if you want to automate at scale, you can write your own assessment agent, define its six evaluation dimensions yourself, and let updates go through automatically when they pass your agent's bar.

And you don't have to guess whether your rules are holding up. You can track weekly F1, precision, and recall across AI reviews, deterministic verifier rules, and PR risk evals. Internally, our Requirement Evals are at 94% F1 (98% precision, 91% recall) across 2,575 cases, and improving.

At the same time, real PR activity turns into new requirements with appropriate scopes, grouped by repo, path, or package, like a claims service, a FHIR integration package, or the code paths that touch PHI. That avoids context rot and keeps standards sharp where they apply, instead of growing into one giant global ruleset, which isn't actually how your company works.

It's the same measure, adjust, and repeat loop teams already trust for deployments, now applied to code review.

Moving Up the Maturity Levels

Nobody should turn on automatic merging on day one. Moving from peer-reviewed to risk-assessed code is a progression:

  1. Level 1: Observe. Turn on risk assessments in comment-and-label mode. Watch them for a week alongside your normal human reviews, and provide feedback when the assessment levels don't feel right, so the system can triage it and suggest adjustments.
  2. Level 2: Calibrate. Add deterministic verifiers for sensitive paths and tune rubric weights per service, so your evals reflect how your team actually thinks about risk.
  3. Level 3: Automate. Once you trust the signal and the eval numbers back it up, turn on the merge gate for low-risk PRs. Let deterministic checks and evaluated AI handle the routine work, while people focus on medium, high, and critical changes.

Coding agents made writing code cheap. Checking it is now the slow part. The teams that ship safely at this speed won't be the ones making people click Approve on every diff. They'll be the ones who put deterministic verifiers, transparent risk scoring, and a feedback loop around what merges.