AI code review without lowering the quality bar

Putting an LLM on pull request review is one of the first ideas a team has when it adopts AI. It is also one of the easiest to get wrong, because it starts from a misunderstanding of what code review is for.

Code review is not only about finding bugs

The most thorough study on the subject, done at Google, shows that code review sustains four things at once: catching defects, keeping style and architecture consistent, spreading knowledge across the team, and building a record of who knows each part of the system.

An LLM contributes well to the first two. It contributes nothing to the last two. Automate the whole review and you trade defects found for knowledge not distributed, and the bill arrives months later, when exactly one person understands the critical module.

Where automated review pays

Three categories pay off immediately:

• Mechanical checks with context. A name that breaks the repository convention, missing error handling on one specific path, a log line leaking sensitive data. The LLM reads the diff and the surrounding file, which the linter does not.
• A first pass on a large PR. Not to approve it, but so the human reviewer arrives already knowing where to look.
• Explaining unfamiliar code. It lowers the cost of reviewing an area you do not know, which is precisely where human review tends to go shallow.

Where it gets in the way

The most common failure mode is not the false positive, it is volume. A bot that leaves fifteen low-value comments on every PR trains the team to dismiss comments without reading them. From then on, the comment that mattered gets ignored too.

The second failure mode is a false sense of coverage. "The bot reviewed it" becomes justification for the human to review less, without anyone deciding that explicitly.

The finding that should change your configuration

A controlled experiment published at CCS 2023 compared people writing code with and without an AI assistant on security-related tasks. Two results, together:

1. Participants with the assistant wrote significantly less secure code on most tasks.
2. Participants with the assistant were more confident their code was secure.

The combination is the problem. It is not only that AI introduces risk, it is that it lowers the suspicion of the person who should be reviewing. Which means the security gate needs to get stricter when a team adopts AI, not looser.

Designing the gate

A design that works in practice separates three levels with different authority:

Level 1, blocking and deterministic. Lint, types, tests, SAST, dependency checks. No LLM here: it has to be reproducible and cannot vary between runs.

Level 2, LLM as advisory reviewer. It comments, it does not block, and it works to an explicit budget: at most three to five observations per PR, ordered by severity. The cap is the important part. It forces prioritisation and protects the team's attention.

Level 3, human with a declared scope. The reviewer approves aiming at domain correctness, architectural decisions and readability for whoever comes next. Spelling out that they do not need to hunt for commas is what makes level 2 useful instead of redundant.

For security requirements, anchor levels 1 and 3 in a verifiable standard rather than in judgement. OWASP's ASVS works well here because it is organised in levels, which lets you demand more of a payment endpoint than of an internal screen.

Measure the gate, not the bot

The metric that matters is not how many comments the bot produced, nor how many were accepted. It is whether the gate is holding defects back from production without stalling delivery. Two series are enough: change failure rate and review time. If failure rate did not drop and review time went up, the gate is charging a toll without delivering safety.

It is also worth watching how much code gets rewritten shortly after landing. Repository analyses have been reporting rising duplication and falling refactoring as assistants spread, and that is a symptom that shows up weeks before it becomes an incident.

What this means for your team

Start advisory and capped: an LLM commenting at most five points per PR, with no blocking power, for a month. Compare change failure rate before and after. Only promote a check to blocking once it has a track record of being right.

And keep human review mandatory on AI-generated code, especially on authentication, authorisation and personal-data paths. The available evidence says that is exactly where confidence rises faster than quality.