How to handle false positives in AI code review

A false positive in AI code review is a finding that describes a problem the code does not have. Every automated reviewer produces some, and language models produce them with convincing explanations, which makes them harder to dismiss than a terse static analysis warning. They matter because of what they do to people: a reviewer that cries wolf trains its readers to stop reading, and the next real bug is skimmed past with the noise. The fix is a triage process that records why each finding was dismissed, plus steady tuning of what the reviewer sees and what it is allowed to block.

This page goes deeper on one part of What is AI code review?: what to do with findings that are wrong.

Why AI reviewers flag problems that aren't there

Most false positives come from the reviewer seeing less than the person who wrote the code.

• Missing context. The reviewer sees the diff and perhaps a few surrounding files. It flags an unvalidated parameter without seeing the middleware that validates every request, or a missing permission check that a router applies one layer up.
• Pattern matching. A string concatenated into something that looks like SQL resembles injection, even when the value is a constant. The model recognises the shape of a common bug and reports it.
• Pressure to find something. A prompt that asks "what is wrong with this change?" invites an answer. If the reviewer is not told that finding nothing is acceptable, it tends to produce findings to fill the space.
• Stale knowledge. The model may flag an API as deprecated, or miss that a library now handles a case by default, because its training data predates the version you use.
• Non-determinism. Two runs over the same diff can produce different findings. A false positive that appears once and not again is common.

None of these mean the reviewer is useless. They mean its findings are claims to be checked, not facts.

The real cost of noise

A false positive costs a few minutes of someone's time to read and dismiss. The larger cost is indirect.

When most findings turn out to be wrong, people adjust. They skim, dismiss in bulk, or stop opening the review at all. At that point the reviewer's true positives are lost too, and the team has a process that looks like review without the effect of one. This is the same alert fatigue that affects monitoring systems and security scanners.

There is an opposite failure. A team that tunes aggressively for silence can end up with a reviewer that rarely finds anything, including real problems. A low false positive rate is only good news if the reviewer still catches the bugs it should. Measure both.

Triage: what each decision means

Triage turns a finding into a recorded decision. The common states, and when to use each:

Two distinctions are worth holding to.

False positive is not won't fix. "This is not a bug" and "this is a bug we accept" are different statements, and an auditor or a future maintainer reading the record needs to know which was meant. Mixing them hides accepted risk.

A dismissal needs a reason. "False positive" on its own is unverifiable. "Input is validated by requireCouponSchema in the route definition" lets someone else confirm it in seconds, and shows up the cases where the dismissal itself was wrong.

Reducing false positives at the source

Triage handles noise after it appears. These changes reduce how much appears.

1. Give the reviewer more context. Include the files the change touches, their direct callers and the related tests. Context is usually the biggest lever.
2. Tell it what normal looks like. Short notes on conventions ("all handlers are wrapped by withAuth", "money is stored in integer cents") prevent the same wrong finding from recurring.
3. Say that an empty answer is valid. Instruct the reviewer that reporting no findings is a correct outcome when nothing is wrong.
4. Ask for evidence. Require a file, a line and a concrete explanation of how the problem occurs, such as the input that triggers it. Vague findings without a location are more often wrong.
5. Define the severities. Write down what critical, high, medium and low mean for your codebase. Without definitions, severity drifts, and false positives end up rated high.
6. Cap the number of findings. A limit pushes the reviewer to report its strongest concerns rather than every possible one.
7. Narrow the categories where needed. If style comments are the main source of noise and you already run a linter, tell the reviewer not to report style.

Thresholds: what a finding is allowed to block

Not every finding needs the same power. A common arrangement is:

• Critical and high block the change until they are fixed or triaged.
• Medium and below are shown to the approver but do not block.

This keeps the cost of a false positive proportionate. A wrong low-severity finding costs a glance. A wrong high-severity finding costs a triage decision, which is the right price for something that would otherwise stop a release.

Treat the threshold as something to tighten over time. If high-severity findings are rarely dismissed as false positives, it may be worth blocking on medium too, at least for sensitive repositories.

Measuring and tuning over time

Decisions about the reviewer should come from the triage record, not from impressions.

• Dismissal rate by category. If most security findings are fixed but most performance findings are dismissed, the performance instructions need work.
• Repeat offenders. The same wrong finding on many changes points to a missing convention note or missing context.
• Sampled audits of dismissals. Now and then, have a second person look at a handful of findings marked false positive. A dismissal can be wrong too, and a pattern of wrong dismissals is a process problem.
• Known-bug checks. Run the reviewer over a few changes that contain bugs you have already found. If tuning made it miss them, you have traded recall for quiet.

How AI CodeOps handles it

In AI CodeOps, the findings review that reads each change's diff must answer with a JSON array of findings, each with a title, severity, category, description, recommendation and usually a file. It records at most 12, and an empty array is a completed review with nothing found. See Multi-agent reviews for how the passes fit together.

Each finding can be set to open, acknowledged, in progress, fixed, won't fix, false positive or duplicate from the session's Findings list. Only open high or critical findings in the open, acknowledged or in progress states block the pull request. Marking one fixed, won't fix, false positive or duplicate ends its block, and the triage is kept with the change and in its evidence pack. Changing a status needs a role with triage permission: owners, admins, developers, members and security admins have it; viewers don't.

A CI job that wants a stricter rule can call the release check with blockingSeverity set to medium. And when a finding is real, Fix starts a session to fix it, which goes through the same review and approval as any other change. Details are in Quality gates and on the AI code review page.