Flaky test

A flaky test is a test that sometimes passes and sometimes fails on the same code, with nothing changed in between. The result depends on something the test doesn't control.

What it is

Common causes:

• Timing: fixed sleeps, races between threads, or a timeout that's too tight on a slow machine.
• Order: tests that depend on state left behind by another test.
• Shared resources: a database, port or file used by tests running in parallel.
• The outside world: network calls, the current date, time zones or random values without a fixed seed.

A test that fails on one machine and passes on another is a related problem, usually an environment difference.

Why it matters

Each flaky failure teaches people that red doesn't mean broken. Teams start re-running until green, and then a real failure gets re-run away too. A gate built on tests is only as trustworthy as the least reliable test in it. AI agents that write tests can add flakiness as easily as coverage, so new tests deserve the same scrutiny as new code.

In AI CodeOps

AI CodeOps doesn't detect or quarantine flaky tests. Its release gate judges each workflow by its latest finished run: a passing re-run clears an earlier failure, while a re-run still in progress or canceled doesn't count as a pass. Test results only count for the version of the code they ran on.