Fail open vs fail closed: how quality gates should handle errors

A gate fails open when an error lets the change through, and fails closed when an error stops it. Quality gates should fail closed: if the test runner crashes, the review times out, the evidence is missing or the gate service cannot be reached, the change is refused until a real verdict exists. A gate that fails open looks fine on normal days and silently switches off on the days something is wrong, which are the days it exists for.

This page expands one property from What are quality gates?.

Where the terms come from

The terms come from physical systems, where a designer decides what state a device takes when it loses power or control. A valve can fail open or closed; an electric door lock can fail unlocked, so people can leave a burning building, or locked, so a vault stays shut. Neither is right in general. The choice depends on which failure is worse.

For a quality gate, the comparison is a delayed merge against an unchecked change reaching production. In almost every case the delay is the smaller cost, so gates fail closed.

How gates end up failing open

Few teams decide to build a gate that passes on error. It happens through small, reasonable-looking choices:

• Errors allowed to pass. A CI step marked "continue on error" or "allow failure" so a flaky tool stops blocking builds. The tool then fails for real, and nothing notices.
• Shell scripts that hide exit codes. A command followed by || true, or a pipeline whose exit status comes from the last command rather than the one that failed.
• Timeouts treated as neutral. A security scan that times out is reported as "neutral" or "skipped", and branch protection does not require it.
• Skipped jobs counted as success. On some CI systems, a job skipped by a condition reports success. A required check can then pass without running.
• Empty output read as clean. A review tool that crashed writes no findings file. The gate reads "zero findings" and passes.
• Unparseable answers ignored. An AI reviewer returns prose instead of the structured result the gate expects, and the gate treats the absence of findings as approval.
• HTTP errors ignored. A CI step calls a gate service, gets a 500 or a timeout, and only fails if the response explicitly says "denied".

Each of these turns "we could not check" into "we checked and it is fine".

Missing evidence is a decision, not a default

Some evidence is legitimately absent. A documentation-only project may have no tests. A new service may not have an SBOM yet. A test command may not print coverage.

The rule is not that every missing input must block. It is that each case is decided on purpose, written down and visible:

Whatever you choose, a missing result must never be displayed as a green tick. "Not checked" and "passed" need to look different to the person approving.

The cost of failing closed, and how to manage it

Failing closed has a real cost: when a dependency of the gate is down, nothing ships. Teams that ignore this cost tend to respond to the first outage by disabling the gate. Plan for it instead.

• Retry before failing. Transient network errors and rate limits should be retried with backoff a few times before the gate reports a failure.
• Say what failed. "Release check unavailable" and "release denied: high-severity finding" are different problems with different owners. Both block; the message tells people what to do.
• Keep a break-glass path. For real emergencies, an authorised person can override the gate for one change, with a recorded reason. The override is rare and visible, and the gate stays on.
• Watch the gate itself. Track how often the gate errors as well as how often it denies. A gate that errors often will be bypassed.

Designing a fail-closed gate

1. Start from deny. The gate's default answer is no. It returns yes only when every required piece of evidence is present and acceptable.
2. Treat errors as denials. Exceptions, timeouts and malformed inputs all produce a denial with a reason, never an approval and never "neutral".
3. Parse strictly. If a tool's output cannot be read, the tool did not succeed.
4. Bind evidence to the version. A test result or approval for an earlier commit does not count for the current one.
5. Make every path cross the gate. If merges on the Git host skip a gate that console merges respect, the gate fails open by another route.
6. Make the caller fail closed too. A CI step that calls the gate must fail when the call fails.

The last point is easy to miss. Here is a CI step that fails closed when calling a release gate over HTTP: any network error, non-success status or unreadable response fails the job, and so does any answer other than an explicit allow.

response=$(curl -sf --max-time 30 -X POST "https://<your-console>/api/release/check" \
  -H "Authorization: Bearer $WORKSPACE_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"pullRequestURL\":\"$PR_URL\"}") || { echo "Release check unavailable: failing closed"; exit 1; }
[ "$(echo "$response" | jq -r '.allowed')" = "true" ] || { echo "$response" | jq -r '.reasons[]?'; exit 1; }

How AI CodeOps does it

AI CodeOps gates are built to fail closed, with a few documented exceptions.

• If the AI findings review fails, times out, runs out of budget or returns something that cannot be read, the change is held: "The automated review of this change didn't complete, so its findings are unknown."
• If the gate check cannot run when a pull request is about to open or merge from the console, the pull request is not opened and the merge does not happen: "The release check couldn't run just now."
• The CI release check returns allowed: false with the summary "Evaluation failed" if the evaluation errors, and denies a change with no matching sessions.
• With disallowedLicenses set, a project with no SBOM is denied, because its licenses cannot be checked.
• If the budget check fails when a session starts, the session is blocked rather than run unmetered.

The exceptions are deliberate and shown rather than hidden. A project whose tests cannot run records no test result, so no test result blocks the change, and the run page shows that it was not verified. A coverage floor set with minCoveragePct only applies when a test run reported coverage. And if the budget check fails during a running session, the session keeps running.

The endpoint returns HTTP 200 for both allow and deny, so read allowed, and it answers 429 above 60 checks a minute for a workspace, which a fail-closed CI step should retry and then fail. See Quality gates for what the gate checks, the API reference for every field, and AI code quality gates for the overview.