Test coverage gates: floors, new-code coverage and their limits

A test coverage gate fails a change when measured test coverage falls below a set floor. It is one of the most common quality gates because it is cheap to compute and easy to understand. It is also one of the most misread: coverage shows how much code ran during the tests, not whether the tests checked anything. Used well, a coverage gate stops test coverage from sliding and nudges every change to come with tests. Used as a target in itself, it produces tests written to raise a number.

This page goes deeper on one kind of evidence from What are quality gates?.

What coverage measures

Coverage tools instrument the code, run the tests and record which parts executed. The common measures are:

• Line or statement coverage. The share of lines or statements that ran at least once. The most widely available.
• Branch coverage. The share of decision outcomes that ran: for each if, both the true and the false path. Stricter than line coverage, since a line containing a condition can run without both outcomes being tested.
• Function coverage. The share of functions called at least once.
• Condition and decision coverage. Stricter forms used in safety-critical software. Modified condition/decision coverage (MC/DC), for example, is required for the most critical level of airborne software under the DO-178C guidance.

Most teams gate on line coverage because every mainstream tool reports it, and add branch coverage where their tools report it reliably.

Kinds of coverage gate

There are three common designs, and they behave quite differently.

A project floor is the easiest to set up and the weakest per change: in a large codebase, an untested change of a few hundred lines may move the total by a fraction of a percent. New-code coverage is the most useful for reviewing individual changes, because it asks a direct question: are the lines this change added exercised by tests?

What coverage cannot tell you

Coverage is a measure of execution, so it has blind spots that matter for a gate.

• Assertions. A test that calls a function and checks nothing still covers every line it runs.
• Correctness of expectations. A test that asserts the wrong value covers the code and passes.
• Missing behaviour. Coverage measures the code that exists. A missing check, such as an absent permission test, has no lines to be uncovered.
• Integration effects. High unit test coverage says little about how components behave together.

There is also a behavioural effect, often summarised as Goodhart's law: when a measure becomes a target, it stops being a good measure. A gate that only rewards the number invites tests written to raise it.

Coverage and AI-written code

Coverage gates are a cheap way to check that an AI agent wrote tests along with its code rather than instead of them. They are also easy for an agent to satisfy without real testing, usually without any intent to deceive: the agent was asked for passing tests and coverage, and it produced them.

Patterns to watch for in AI-written tests:

• tests that call code and assert only that no exception was thrown;
• snapshot tests that record whatever the code returns as the expected value;
• mocks that replace the logic under test, so the test checks the mock;
• tests generated from the implementation, which repeat its bugs as expectations.

Treat coverage as a floor that every change must clear, and the test diff as something a reviewer reads. See How to review code written by an AI agent for what to look for.

Better signals to pair with coverage

• Mutation testing. A tool makes small changes to the code, such as flipping a comparison, and checks whether any test fails. Surviving mutations reveal code that runs under test but is not really checked. Tools exist for most languages, such as Stryker for JavaScript and TypeScript, PIT for Java and mutmut for Python. They are slower than coverage, so teams often run them on a schedule or on changed files only.
• Regression tests for fixes. A bug fix should include a test that fails without the fix. That test proves something coverage cannot.
• Review of the test diff. Deleted tests, loosened assertions and changed expectations are visible in the diff and invisible in a coverage number.

Setting a coverage floor

1. Measure first. Run coverage on the main branch and note the current figure for each project.
2. Choose the gate type. New-code coverage if your tools support it; otherwise a project floor plus a ratchet.
3. Set the floor near today's number. A floor the codebase already meets is enforceable from day one.
4. Exclude deliberately. Generated code, vendored code and test helpers can be excluded, in configuration that is reviewed like code. Silent exclusions are a way to game the gate.
5. Decide what happens with no coverage figure. If a test run does not report coverage, the gate either blocks or skips the floor and says so. Do not show it as passing.
6. Raise it over time. Review the floor periodically and raise it when coverage has risen.

How AI CodeOps does it

When an AI CodeOps session runs your project's tests in its sandbox, it reads line coverage from the test output when the output carries a summary: a pytest-cov TOTAL line, an Istanbul All files row (as printed by Jest and Vitest), or a labelled line such as coverage: 85.3%. The figure is stored with the test run.

The coverage floor is the minCoveragePct option of the release check your CI calls:

{ "pullRequestURL": "https://github.com/acme/checkout/pull/42", "minCoveragePct": 80 }

With it set, the most recent test run that reported coverage must meet the floor, or the release is denied with a reason such as "Coverage 71% is below the required 80%". If no test run reported coverage, the floor is not applied, so make your test command print a summary, for example pytest --cov or vitest run --coverage, if you want it to bite. The per-session gate in the console blocks failing tests but does not check coverage. Failing tests deny the release whatever the coverage.

See the coverage floor entry for more detail, the API reference for every release check option, and AI testing for how tests run.