FLAKY TEST FINDER · GUIDE
"Flaky" means a test fails sometimes and passes other times with no code change in between. It is different from a broken test, which fails every time — see the first question below for why that distinction matters before you start debugging anything.
Guessing from memory ("I think it's the checkout test") wastes time on the wrong one. Compare two or more runs and let the pass/fail pattern tell you — free, runs in your browser, nothing uploaded.
Open Flaky Test Finder →CI runners are usually slower and more resource-constrained than a laptop, and often run tests in parallel across more workers. That surfaces timing assumptions (a fixed sleep instead of an actual wait condition), shared state between tests running concurrently, and real network or filesystem calls that a local run's warm cache or already-open connection was hiding.
A broken test fails every time under the same conditions — that is a real regression in the code or the test. A flaky test fails sometimes and passes other times with no code change in between — the test or its environment is nondeterministic. Treating a broken test as flaky (retrying until green) ships a real bug; treating a flaky test as broken (blaming the last commit) wastes time bisecting a change that did nothing.
As a short-term unblock for a release, sometimes. As a permanent policy, no — a suite where flaky tests are silently retried stops being a signal at all, since a real regression that also happens to race can pass on retry too. Track which tests are flaky and fix the highest-value ones; a rerun-until-green CI step should be a visible, counted exception, not silent default behavior.
Run the suite (or the reruns your CI already does) at least twice and compare pass/fail per test — a test that flips is flaky, a test that fails every time is a real bug. The free Flaky Test Finder tool on this site does exactly that from JUnit XML reports you already have, with no signup and nothing uploaded.
A fixed sleep(1) instead of waiting for an actual condition (an element to appear, a job to finish, a lock to release) works on a fast machine and fails on a loaded CI runner. Look for hardcoded delays and replace them with a poll-until-true or an explicit wait on the real event.
A global variable, a database row, or a file on disk that one test writes and does not clean up can make a later test pass or fail depending on execution order or whether tests run in parallel. This is the most common cause when a test only fails when run alongside the full suite, never in isolation.
A test that hits a real API, a real DNS lookup, or a real disk write inherits that dependency's own reliability — a slow upstream service, a rate limit, or a transient DNS failure will show up as an intermittent test failure that has nothing to do with your code. Mock the boundary in unit tests; keep real calls to a smaller, explicitly-flagged integration suite.
A test suite that assumes it always runs in file order, or that test IDs are assigned sequentially, breaks the moment a test runner shards or randomizes execution — which most CI systems now do by default to cut wall-clock time. Randomize test order locally on purpose to surface this before CI does.