A payments team at a mid-sized SaaS company had a release process that looked disciplined on paper. Every pull request ran a suite of 2,400 automated tests. Merges to main required a green build. Releases required a second green build on the release branch. Nobody could point to a single rule that had been violated, and yet the team had shipped three billing bugs to production in the previous two quarters that their test suite should have caught — and, on closer inspection, had caught. The tests had failed. They had also, on rerun, passed. Engineers had stopped reading which tests failed on the first attempt, because on that team, "failed once" and "passed eventually" were treated as the same outcome. The build was green. The bug shipped anyway.

This is not a story about bad engineers or an underfunded QA function. It is a story about what happens when an organization stops asking whether its test suite is still measuring anything. A green checkmark is only useful if it means something specific and stable: that the code under test behaved correctly against a known set of conditions. The moment a team starts rerunning failures until they pass, without recording why they failed, that checkmark stops meaning that. It starts meaning something closer to "this code passed at least once, eventually, under conditions we did not write down." Most engineering organizations never make a conscious decision to accept that weaker guarantee. It happens by accretion, one --reruns 3 flag and one "just retry it, it's flaky" Slack message at a time, until the team is release-gating on a signal nobody would defend if asked to justify it from first principles.

This article treats flaky tests as what they actually are: a measurement-validity problem. It lays out a root-cause taxonomy for diagnosing them, explains the statistical mechanism by which retry-until-green quietly redefines "passing," and provides a concrete framework — ownership models, a flakiness budget, a triage checklist, and build-versus-buy guidance — for treating test reliability as a governed engineering discipline rather than a background irritation that gets silenced with a decorator.

Why "It's Just Flaky" Is a Statistical Claim, Not an Excuse

When an engineer says "ignore that test, it's just flaky," they are making an implicit statistical claim: that the test's failure on this run is uncorrelated with the code change under review, and that its underlying pass rate is high enough that a rerun is a reasonable way to confirm the code is fine. That claim is sometimes true. It is frequently not checked at all.

Consider what a CI pass rate is actually supposed to represent. If a test suite has a true pass probability of 99.9% against correct code, and a build requires all N tests to pass, the naive expectation is that N=2,000 tests would produce a build failure rate driven almost entirely by genuine regressions. But if even 2% of those 2,000 tests have a 5% intrinsic flake rate — failing occasionally regardless of code correctness — the probability that at least one of those 40 tests fails on any given run climbs quickly, and a team facing that failure rate every few builds develops a habit: rerun and move on. Once that habit exists, it does not stay confined to the 2% of genuinely flaky tests. Engineers under deadline pressure cannot always distinguish "this is one of the 40 known-flaky tests" from "this is a real regression I don't feel like investigating," so the rerun reflex generalizes. The team's operational definition of "the test suite passed" silently shifts from "every test passed on the first attempt" to "every test passed on some attempt within N reruns," and nobody voted on that change.

The Arithmetic of Compounding Flakiness

The mechanism above is easier to see with actual numbers rather than intuition alone. Suppose a suite has 3,000 tests, and 95% of them are perfectly deterministic — they only fail when the code under test is actually wrong. The remaining 5%, or 150 tests, each carry a modest 2% chance of failing on any given run regardless of code correctness. That 2% figure sounds negligible for any single test. It is not negligible in aggregate.

The probability that a specific flaky test passes on a given run is 0.98. The probability that all 150 flaky tests pass simultaneously on a single run, assuming independence, is 0.98 raised to the 150th power, which works out to roughly 4.8%. In other words, on a suite with these characteristics, a build has only about a one-in-twenty chance of getting through the flaky subset cleanly on the first attempt, even when every line of code under test is completely correct. Widen the flaky fraction slightly, to 8% of a 3,000-test suite with the same 2% per-test flake rate, and the probability of a fully clean first-attempt run against correct code falls under 1%.

This is not a hypothetical edge case; it is the ordinary arithmetic of any suite with a nontrivial number of independently flaky tests, and it explains something that otherwise looks irrational: teams do not adopt retry-on-failure because they are lazy or undisciplined. They adopt it because, given the numbers above, a policy of investigating every first-attempt failure would mean investigating dozens of false alarms for every real regression, which is not a sustainable use of engineering time without better tooling to separate the two. The retry reflex is a rational adaptation to a suite that has already crossed a flakiness threshold. The mistake is not retrying. The mistake is retrying blindly, without recording which tests are being retried, how often, and why — which is the difference between a bounded, logged mitigation and an unbounded erosion of the release signal.

Note: this table is an illustrative mathematical demonstration using the standard independence assumption for compounding probabilities, not a measured figure from any specific organization's test suite. Real suites often show correlated rather than fully independent flakiness — for example, several tests sharing one flaky fixture fail together — which changes the exact numbers but does not change the underlying dynamic that even a small flaky fraction compounds quickly across a large suite.

This matters because release decisions — deploy or hold, hotfix or wait, block the release train or let it through — are downstream of that signal. A measurement instrument whose calibration drifts without anyone tracking the drift is not a minor inconvenience in a laboratory; it is why a scale that reads two pounds light doesn't get flagged until someone weighs something on a second scale and notices the discrepancy. CI pipelines rarely have a second scale. The build is green, or it is not, and once "green" has been redefined by an accumulation of retry logic, there is no external check catching the redefinition.

Google's engineering organization has published unusually specific numbers on this. In a widely cited 2016 post from the Google Testing Blog, engineer John Micco reported that at Google, roughly 1.5% of all test runs report a flaky result, that almost 16% of the company's tests show some level of flakiness, and — the number that matters most for release-gating — that approximately 84% of the transitions observed from a passing build to a failing build involve a flaky test rather than a genuine regression (Google Testing Blog, 2016). Read that last figure carefully. It does not say 84% of test failures are flaky. It says that when a previously green signal turns red, the flaky-test explanation dominates over the genuine-regression explanation by a wide margin. If that ratio holds even loosely in a smaller organization's pipeline, the rational response to "the build went red" is not "something broke," it is "probably nothing broke, but check." That is a fundamentally weaker signal than most teams believe they are operating with, and it is why so many engineers develop the rerun reflex in the first place — it is often the statistically correct guess, which is exactly what makes it dangerous to apply uniformly.

Spotify's engineering team reached a related conclusion from a different angle. In a 2019 post describing their approach to test flakiness, they reported a baseline flakiness rate of 6% across their test suite, and — notably — that simply making flakiness visible to engineers through a metrics dashboard, without any other intervention, reduced that rate from 6% to 4% within two months (Spotify Engineering, 2019). No retry policy changed. No dedicated flakiness team was created in that window. The rate dropped because engineers could see it, and visibility alone shifted behavior. That single data point is worth dwelling on because it identifies the actual failure mode this article is about: flakiness is not primarily a technical problem that resists solutions. It is primarily an invisible problem that never gets prioritized because nobody is measuring it, and the moment it becomes visible, the organization's existing engineering discipline starts addressing it without needing new tooling.

A Root-Cause Taxonomy of Flaky Tests

Treating flakiness as a defect category requires being able to classify it, the same way a bug tracker classifies defects by severity and subsystem rather than lumping everything into "broken." The following taxonomy groups flaky test root causes by where the nondeterminism actually originates, because that determines both who should fix it and how.

Test-order dependency

A test passes in isolation and fails when run after a specific other test, because it implicitly relies on state that an earlier test happened to leave behind — a database row, a cache entry, a mutated global variable, a file on disk. These tests are often invisible for months because CI runs preserve a stable test order, until someone reorders the suite for parallelization, introduces test sharding, or a new test gets inserted alphabetically before the dependent one, and a previously "stable" test starts failing intermittently in a way that looks unrelated to any code change.

Shared mutable state across parallel workers

As suites grow, parallel execution becomes necessary for CI runtime to stay tolerable. Parallelization exposes any test that mutates shared state without isolation — a shared test database, a shared temp directory, a shared in-memory singleton, a shared port. Two tests running concurrently against the same seeded database row will intermittently collide, and the failure will not reproduce reliably locally because local runs are usually sequential.

Timing and race conditions inside the test itself

This is distinct from a race condition in the production code under test. Here, the application logic may be entirely correct, but the test asserts on a UI element, an async callback, or a background job's output before that side effect has actually completed. A sleep(2) that usually works becomes a source of failure the moment CI runner load varies, a downstream service responds 200ms slower than usual, or the test runs on a different, slower instance type. This category is enormous in end-to-end and UI test suites and is one of the most commonly documented causes across large-scale industrial studies of flaky tests.

Unmocked or under-mocked external dependencies

A test that makes a real network call to a third-party API, a real DNS lookup, or a real call to a rate-limited internal service will intermittently fail for reasons that have nothing to do with the code under test — a transient 503, a rate limit, a DNS hiccup, a certificate rotation on the dependency's side. These failures are genuinely external and genuinely nondeterministic from the test's perspective, but they still corrupt the CI signal exactly the same way internal flakiness does, because the person reading a red build cannot tell the difference without investigating.

Hypothetical example: the notification test that depended on a real vendor sandbox. Consider a hypothetical B2B SaaS company whose integration test suite includes a test verifying that a user receives an email notification after completing an account setup flow. Rather than mocking the transactional email provider, the test called the provider's sandbox API directly and polled a test inbox for the message to arrive, with a 15-second timeout. For most of a year, this test failed intermittently, roughly 3 to 4 times a week, always attributed informally to "the email vendor being slow sometimes." Nobody had actually confirmed that explanation. When an engineer finally instrumented the test to log the vendor API's response time on every run, the data showed something different: the sandbox environment's median response time was well within the timeout, but its 95th-percentile response time frequently exceeded 15 seconds during the vendor's own peak traffic hours, which happened to overlap with this company's own peak CI usage hours because both companies were headquartered in the same time zone. The fix was not a longer timeout, which would have masked the same problem less visibly. It was replacing the real sandbox call with a contract-tested mock for the unit and integration layer, and moving the one genuinely valuable end-to-end vendor check to a separate, non-blocking nightly job with its own generous timeout and its own alerting, so that a slow third party could no longer intermittently gate every engineer's pull request.

Async and wait misuse

Related to timing races but distinct enough to warrant its own category: tests that use fixed-duration waits (sleep, Thread.sleep, arbitrary timeouts) instead of polling for a specific condition or subscribing to a completion event. This is one of the most consistently identified root causes in academic literature on flaky tests, including in Microsoft Research's own industrial studies, which found asynchronous-call handling to be a leading cause of flakiness in Microsoft's internal test suites and built a dedicated tool, described as a "Flakiness and Time Balancer," specifically to address async-wait-related flakiness in their pipelines (Microsoft Research, "A Study on the Lifecycle of Flaky Tests").

Environment nondeterminism

Tests that depend on system clock behavior across time zones or daylight saving transitions, locale-dependent string formatting, floating-point precision that varies by CPU architecture, or randomized input generation without a fixed seed. These often surface only intermittently because the triggering condition — a specific date, a specific locale, a specific hardware target — does not occur on every run.

Test data collisions under parallel execution

Distinct from shared mutable state in that the tests may be properly isolated in code but collide on external identifiers: two parallel test runs both generating a user with the same email address, the same idempotency key, or the same order ID because the test data factory uses a predictable or insufficiently unique generator. This category has grown as more teams adopt aggressive test parallelization to control CI runtime.

Resource exhaustion under load

Tests that pass reliably in isolation begin failing under full parallel CI load because the test runner is memory-constrained, connection pools are exhausted, or container CPU throttling introduces latency that trips a timeout that was calibrated against a quieter environment. These are frequently misdiagnosed as "flaky" in the code sense when the actual defect is in CI infrastructure capacity planning.

The table below summarizes this taxonomy along with its typical detection signature and primary owner, which the ownership section later in this article builds on directly.

The Trust-Erosion Mechanism: How a Red Build Stops Meaning Anything

Flaky tests do not damage a CI pipeline through the individual failures. They damage it through the adaptive behavior engineers develop in response to those failures, and that behavior compounds in a predictable sequence.

Stage one: selective rerun. An engineer sees a red build on an unrelated test, reruns the job, and it passes. This is individually reasonable. Repeated across a team and a codebase, it becomes the default response to any red build, including ones that are not actually flaky.

Stage two: institutionalized retry. The team stops relying on individual judgment and encodes the rerun behavior into CI configuration — a rerun-on-failure step, a pytest-rerunfailures flag applied suite-wide, a CI vendor's built-in "retry failed jobs" feature applied without discrimination. At this point, the redefinition of "passing" is no longer a habit; it is policy, and it applies uniformly to genuinely flaky tests and to tests that failed because of a real regression that happened to be transient — a null pointer triggered only under specific request timing, for instance.

Stage three: quarantine sprawl. As flaky tests accumulate faster than anyone fixes them, teams introduce a quarantine mechanism — a skip tag, a separate "known flaky" CI job that does not block merges, an xfail(strict=False) marker. This is a legitimate tool used correctly. Used as a permanent parking lot, it becomes a one-way door: tests go in, almost none come out, and the quarantined suite grows to the point where a meaningful fraction of the organization's total test investment is producing no release-gating signal at all while still consuming CI compute and maintenance attention.

Stage four: alert fatigue and selective blindness. Once quarantine lists and retry counts grow large enough, engineers stop reading CI failure output carefully at all. A red build triggers a reflexive rerun rather than an investigation, even for tests that are not on any quarantine list, because the base rate of "it's probably nothing" has been trained into the team by stages one through three. This is the point at which the original argument of this article becomes concrete: the team has stopped treating red as a meaningful signal, and the shift happened through a sequence of locally reasonable decisions, none of which was ever evaluated as a policy change to what "passing" certifies.

Stage five: real regressions ship. Somewhere in the noise, a genuine regression produces a test failure that looks exactly like the hundreds of flaky failures the team has learned to dismiss. It gets rerun, passes on retry because the underlying bug is itself intermittent (a race condition, a timing-sensitive data corruption, an off-by-one under specific load), and ships. This is precisely the mechanism in the payments team scenario that opened this article: the tests that caught the billing bugs did their job. The organizational process built around them had stopped listening.

Hypothetical example: quarantine sprawl at a growing fintech platform. Consider a hypothetical fintech company whose engineering team grew from 15 to 90 engineers over two years, during which its test suite grew from roughly 800 tests to just over 9,000. Early in that growth, a single engineer had introduced a lightweight @quarantine tag for tests that were failing intermittently and blocking merges, with the informal expectation that quarantined tests would be revisited "soon." No expiration mechanism existed, and no dashboard tracked the list's size. Eighteen months later, an audit prompted by a near-miss incident (a quarantined test that, on inspection, had been silently catching a real intermittent bug in a fee-calculation service for four months) found 340 tests under the quarantine tag, of which 210 had not been touched in over ninety days, and for which the original author was still identifiable in only about a third of cases. Nobody had decided, as a matter of policy, that 340 tests — roughly 3.8% of the entire suite — would run without contributing to the release-gating signal. It had happened one merge-deadline exception at a time, and the list had simply never been reviewed as an aggregate quantity until an incident forced the question. The remediation was not a tooling purchase; it was a two-sprint effort to work through the backlog, delete tests whose coverage value no longer justified the maintenance cost, fix the ones that mattered, and replace the open-ended tag with the time-boxed, owner-linked quarantine format described later in this article.

Hypothetical example: the "flaky" checkout test that wasn't

Consider a hypothetical mid-sized e-commerce platform whose checkout integration test suite includes a test verifying that inventory is correctly decremented when an order is placed. The test had a documented history of intermittent failure, attributed by the team to test-order dependency from a shared test database, and it carried a @pytest.mark.flaky(reruns=3) decorator applied eighteen months earlier by an engineer who has since left the company. In practice, the test was failing roughly once every twelve runs because of a genuine race condition in the inventory service: under specific timing, two near-simultaneous checkout requests for the last unit of a limited-stock item could both read available inventory as one before either write committed, allowing both orders to succeed. The rerun almost always passed, because the timing window was narrow and the CI environment rarely reproduced the exact race. Over eighteen months, the decorator suppressed the signal on every occasion the bug actually manifested in the test environment. The bug reached production during a flash sale, where request volume made the timing window far more likely to trigger, and a batch of oversold orders had to be manually resolved with customers. Nothing about this scenario required the checkout service's concurrency bug to be unusual or the test environment to differ from production in any meaningful way — it required only that a flaky-test suppression mechanism had been applied without anyone ever separating "this fails because of shared test-database ordering" from "this fails because of a real, intermittent production bug." Both produce the identical symptom: intermittent test failure. Only a root-cause investigation distinguishes them, and the retry decorator was specifically the mechanism that made that investigation feel unnecessary.

Quarantine as a Legitimate Tool, Used Correctly

None of the above is an argument against quarantine as a mechanism. A time-boxed, owner-linked quarantine is one of the more useful tools available for keeping a release pipeline unblocked while a real fix is worked out, and refusing to use it at all, on principle, tends to produce its own failure mode: engineers under deadline pressure reach for an undocumented workaround instead, such as commenting out an assertion or deleting the test outright without recording why, which destroys the coverage entirely rather than temporarily suspending it. The distinction that matters is not quarantine versus no quarantine. It is bounded, visible, accountable quarantine versus open-ended, invisible, unowned quarantine.

A quarantine entry that meets a reasonable bar for legitimacy typically includes four elements: a linked ticket describing the suspected root cause, even if that description is provisional; a named owner, ideally a team rather than an individual who may leave; an expiration date after which the test is either fixed, re-evaluated, or deliberately deleted, not silently re-quarantined for another indefinite period; and visibility in whatever dashboard or report the broader engineering organization actually looks at, not only in the test framework's internal configuration where it is invisible to anyone who does not go looking for it. A quarantine mechanism satisfying all four criteria functions the way a well-run incident postmortem process functions: it acknowledges a known gap explicitly, assigns responsibility for closing it, and creates a natural forcing function (the expiration date) that prevents the gap from becoming permanent by default.

The failure pattern described earlier in the fintech quarantine-sprawl example did not happen because quarantine itself was the wrong tool. It happened because the tool was missing three of those four elements from the outset, and nobody noticed the gap until an incident forced a retrospective audit. A useful gut check for any team currently using an informal skip-tag or quarantine mechanism: pull up the full list of currently quarantined or retry-enabled tests right now, and check how many entries are missing an owner, a reason, or an expiration date. Most teams that have never done this exercise are unpleasantly surprised by the result.

Measuring Flakiness Properly

An organization cannot manage what it does not measure, and "we know we have some flaky tests" is not a measurement. A workable measurement approach needs at minimum three figures, tracked over time and visible to the engineering organization, not buried in a CI vendor's dashboard nobody opens.

Flakiness rate. The proportion of test runs that produce a different result than a prior run against the identical commit, typically calculated per-test and rolled up per-suite over a trailing window. CircleCI's Test Insights feature, for example, defines a flaky test operationally as one that "failed and passed on the same commit" within a rolling observation window, and surfaces a flaky-test count and success rate per test directly in its dashboard (CircleCI documentation). The specific window length matters less than having one consistently applied definition the whole organization uses, rather than each team eyeballing "seems flaky to me."

Quarantine aging. For every test placed in quarantine (skipped from release gating, retried automatically, or marked xfail), track how long it has been quarantined and who owns it. A quarantine list with no aging visibility becomes, by default, a permanent exemption list. A quarantine list with aging visibility becomes a queue that someone can be held accountable for working down.

Mean time to fix (MTTF) for flaky tests. From the moment a test is first flagged as flaky to the moment it is either fixed (returns to reliable pass/fail behavior tied to actual code correctness) or deliberately deleted, measured in days. This is the single most useful trend line for evaluating whether an organization's flakiness governance is working, because a stable or falling MTTF indicates the team is actually retiring flaky tests rather than accumulating them.

Chart 1: Illustrative flakiness rate trend under two governance approaches

Chart type: Line chart, two series over time.

Axes: X-axis = weeks since a flaky-test governance program begins (Week 0 through Week 16). Y-axis = suite-wide flakiness rate (percentage of test runs producing an inconsistent result versus the same commit).

Underlying data (illustrative and not measured industry benchmarks — these figures are a hypothetical scenario constructed to demonstrate the shape of the effect, not real measured data from any company):

What this shows: Under a rerun-and-ignore approach, flakiness tends to compound as the test suite grows, because new flaky tests are added faster than any are fixed and there is no countervailing pressure. Under a measured and owned program, visibility and accountability drive the rate down and keep it down. The specific numbers here are illustrative, but the direction is consistent with what Spotify reported from a real intervention: visibility alone, without any other change, moved their measured flakiness rate from 6% to 4% in two months (Spotify Engineering, 2019). The chart above generalizes that real single data point into an illustrative multi-week trend to show the pattern such an intervention typically produces, not to claim it as measured data itself.

Chart 2: Where pass-to-fail transitions actually come from

Chart type: Horizontal bar chart, single series.

Categories (Y-axis): Source of a pass-to-fail transition on a previously green build. Values (X-axis): Share of observed transitions.

Underlying data (real, sourced):

Source: Google Testing Blog, "Flaky Tests at Google and How We Mitigate Them," John Micco, May 2016 — https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html

What this shows: At Google's scale and with Google's specific test suite composition, the large majority of red-to-green-to-red transitions on unchanged code were attributable to flaky tests rather than actual regressions. This is a single, vendor-reported (in this case, company-reported) figure from one organization's internal measurement at a specific point in time, not a universal constant — smaller organizations with smaller, younger test suites may see a different ratio. It is included here because it is the most concrete, publicly available figure demonstrating why "the build went red" cannot be treated as strong evidence of a regression without further investigation, and because it explains, mechanically, why engineers develop a rerun reflex: at an 84/16 split, guessing "probably flaky" is usually correct, which is exactly the trap this article is describing.

Where Flakiness Metrics Themselves Get Misread

Introducing a flakiness rate, a quarantine-aging figure, and an MTTF trend line is a meaningful improvement over having no metric at all, but the metrics themselves are not immune to the same measurement-validity problem this article opened with. Three specific misreadings are common enough to name explicitly.

Confusing a low flakiness rate with a healthy suite. A suite can show a low aggregate flakiness rate simply because most of its genuinely unreliable tests have already been quarantined out of the denominator. If quarantined tests are excluded from the flakiness-rate calculation entirely rather than tracked as their own aging metric, the headline number improves precisely by hiding the problem rather than solving it. Any flakiness rate reported without a companion quarantine-size figure should be treated as incomplete.

Treating small-sample noise as a trend. A test that runs only a handful of times per week — common for slower integration or end-to-end suites that do not run on every pull request — can show a dramatic-looking flakiness rate shift (from 0% to 20%, for instance) off a single additional failure, purely because the sample size is small. Rolling this kind of test into the same weekly trend line as tests that run hundreds of times per day, without weighting for run count, produces a noisy and occasionally misleading top-line number. A more reliable practice is to report flakiness rate alongside run count per test, and to treat any test with fewer than roughly twenty runs in the observation window as provisionally classified rather than confidently flaky or confidently stable.

Mistaking infrastructure incidents for a flakiness trend. A spike in the flakiness rate that coincides with a specific CI runner outage, a cloud provider incident, or a shared staging database going down is an infrastructure event, not evidence that the test suite itself has become less reliable. Teams that do not annotate their flakiness trend line with known infrastructure incidents will periodically misinterpret a one-time external event as a structural regression in test quality, and can end up chasing a root cause in test code that never existed, because the actual cause was a five-hour cloud provider degradation that has nothing to do with any individual test's design.

Each of these failure modes has the same underlying shape as the core argument of this article: a number that looks authoritative on a dashboard can still misrepresent what it claims to measure, unless someone is actively checking the number against what it is supposed to certify.

What Flakiness Actually Costs

Engineering leaders evaluating whether to invest time in flakiness governance often ask, reasonably, what the return on that investment looks like. The honest answer is that few organizations have precise figures, because the cost of flakiness is distributed across many small interruptions rather than concentrated in one visible line item, which is itself part of why it stays underinvested. The illustration below is not measured data from any real organization; it is a simple, transparent model an engineering leader can substitute their own figures into.

Assume a 40-engineer team, an average fully loaded engineering cost of $90 per hour, and a suite where 5% of pull requests are blocked by a flaky failure requiring at least a rerun-and-wait cycle, at an average cost of 12 minutes of engineer attention per incident (context-switch back into the failing build, decide whether to rerun or investigate, wait for the rerun, resume prior work). If the team merges roughly 25 pull requests per engineer per month, that is 1,000 pull requests per month across the team, of which 50 trigger a flaky-failure interruption.

These figures are an illustrative model, not measured data from any real company or QAtronic engagement. The point of the model is not the specific dollar figure, which will vary widely by team size, wage base, and actual flaky-blocked PR rate — it is the structure of the calculation, which an engineering leader can rerun with their own numbers.

Two things make this model conservative rather than alarmist. First, it counts only the direct interruption cost of a rerun-and-wait cycle, not the harder-to-quantify cost of a real regression that ships because it was misclassified as flaky noise, which is typically far larger per incident but rarer and therefore easy to discount in a simple monthly model. Second, it does not account for CI compute cost, which scales directly with rerun volume — a suite that reruns 5% of pull requests an average of two extra times is running meaningfully more CI compute than its nominal pass-rate would suggest, a cost that shows up on a cloud bill rather than an engineering-hours ledger and is therefore rarely connected back to flakiness at all. Even a deliberately conservative model like this one is usually enough to justify a modest, ongoing investment in measurement and triage discipline, particularly once an organization crosses the scale-up threshold where suite size and parallelism start pushing the flaky fraction upward on their own.

A Diagnostic Framework for Triaging a Newly Flaky Test

When a test starts failing intermittently, the temptation is to reach immediately for a rerun flag. The following checklist is designed to force a five-minute root-cause pass before any suppression mechanism is applied, because the five minutes spent here is what separates "we manage flakiness" from "we hide flakiness."

1. Reproduce in isolation. Run the failing test alone, several times, outside the full suite. If it passes reliably alone but fails intermittently in the full run, suspect test-order dependency or shared state — not a code defect in the system under test.
2. Check correlation with parallelism level. Rerun the suite at different parallelism settings. If failure frequency scales with worker count, suspect shared mutable state or resource exhaustion, not test logic.
3. Inspect the failure diff, not just the failure count. Look at what actually differed between the passing and failing run — a different response payload, a different timing, a different generated ID. A test that fails with the identical assertion message every time is a different animal from one that fails with a different message each time; the former often points to a real intermittent defect, the latter more often points to test infrastructure noise.
4. Rule out external dependencies. Check whether the test makes any real network call, DNS lookup, or call to a rate-limited service. If so, verify whether the failure correlates with known incidents or rate-limit windows on that dependency.
5. Check the assertion's timing assumptions. Look for fixed sleeps, arbitrary timeouts, or assertions made immediately after triggering an async operation. This is the single most common root cause across published industrial studies of flaky tests and should be checked before any other hypothesis is ruled out.
6. Determine whether this could be a real, intermittent production defect. Explicitly ask whether the system under test could plausibly exhibit the same nondeterminism the test is exhibiting — a race condition, a timing-dependent data corruption, a resource contention issue — under production-realistic concurrency. This step exists specifically because of the checkout-test scenario above; skipping it is how real bugs get quarantined alongside genuinely flaky tests.
7. Classify using the taxonomy, and assign an owner based on the classification (see the ownership section below).
8. Only after steps 1–7, apply a time-boxed quarantine if the fix is not immediate — with an expiration date, an assigned owner, and a linked ticket, not an indefinite skip.

This sequence matters more than any individual step. The organizations that manage flakiness well are not the ones with the cleverest retry tooling; they are the ones that have made steps 1 through 7 a habitual, low-friction part of triage rather than an optional extra that gets skipped under deadline pressure.

Ownership: Who Actually Fixes a Flaky Test

Flaky tests fail to get fixed for the same reason most unowned engineering debt fails to get fixed: no individual or team has clear, durable accountability for them, so they default to whoever happens to notice next, which in practice means nobody consistently. Three ownership models are common, each with real trade-offs.

Author ownership. The engineer who wrote the test, or whoever most recently modified it, is responsible for triaging and fixing it when it flakes. This scales naturally with team size and keeps context close to the fix, since the author usually understands the test's intent. It breaks down when the original author has left the team or company, when the test was auto-generated or copied from a template, or when the flakiness stems from shared infrastructure (a test database, a CI runner class) that no single test author controls.

Code-path ownership. The team that owns the production code path under test is responsible for the test's reliability, on the reasoning that they have the deepest context on whether an intermittent failure reflects a real defect. This model handles the checkout-test scenario well, because the team that owns the inventory service is best positioned to recognize a race condition dressed up as flakiness. It requires clear code ownership mapping (via a CODEOWNERS file or equivalent) to function, and it can create friction when a test spans multiple services or when the flakiness is purely a test-infrastructure artifact unrelated to the code path's actual behavior.

A dedicated test-reliability function. Larger organizations sometimes staff a rotating or permanent role — often called a test infrastructure team, a build cop rotation, or a test reliability engineer — with authority to quarantine tests, track flakiness metrics, and escalate unowned flaky tests back to the responsible team with a deadline. This model is the only one of the three that reliably prevents quarantine sprawl, because it creates a single point of accountability for the aggregate health of the suite rather than relying on distributed goodwill. It is also the most expensive to staff and is usually not justified below a certain scale.

None of these models works without the measurement layer described earlier. Ownership without visibility just distributes the blindness; a code-path team cannot fix what it does not know is flaky, no matter how clearly the org chart assigns responsibility.

A Flakiness Budget: What "Healthy" Actually Looks Like

Borrowing the logic of an error budget from site reliability engineering, a flakiness budget gives a concrete, negotiated answer to a question most teams never explicitly ask: how much nondeterminism in the test suite is acceptable before it triggers action, rather than being tolerated indefinitely.

A workable flakiness budget has three components:

• A ceiling on suite-wide flakiness rate, expressed as a percentage of test runs producing an inconsistent result, reviewed on a rolling basis (weekly or biweekly, not quarterly — flakiness compounds faster than quarterly review cycles can catch).
• A ceiling on quarantine list size and age, expressed as both a maximum count of currently quarantined tests and a maximum age before a quarantined test must be fixed or permanently deleted (not left in limbo).
• A trigger condition, defining what happens when either ceiling is breached — commonly, a temporary freeze on new quarantine additions until the backlog is worked down, similar to how an SRE error-budget breach triggers a feature-velocity freeze in favor of reliability work.

What counts as a reasonable ceiling varies by suite composition and by how release-critical the suite is, and no universal number applies across every organization. What matters is that the number is explicit, tracked, and owned, rather than implicit and unmonitored. A team with a 3% flakiness rate that knows it is 3%, trends it weekly, and has a defined response when it crosses 5% is in a fundamentally healthier position than a team with a 1% rate nobody is tracking, because the second team has no early warning before their rate silently becomes 8%.

Build Versus Buy: Tooling for Flaky Test Detection

Once an organization decides to measure and manage flakiness deliberately, it needs tooling to do so at a scale beyond manual spreadsheet tracking. The landscape splits roughly into language-level retry mechanisms, CI-vendor-native detection features, and dedicated third-party flaky-test platforms.

Language and framework-level mechanisms. Python's pytest-rerunfailures plugin allows marking specific tests as flaky with a controlled, bounded rerun count (@pytest.mark.flaky(reruns=3, reruns_delay=2)), which is materially different from a suite-wide blanket rerun because it requires a deliberate, per-test decision and keeps a record of which tests carry that designation (pytest-rerunfailures documentation). The official pytest documentation itself explicitly names test-order dependence, insufficiently isolated global state, thread-unsafe use of pytest primitives across spawned threads, and overly strict floating-point assertions as common causes, and recommends pytest.approx() for numeric tolerance and dedicated plugins like pytest-randomly specifically to expose order-dependent flakiness during development rather than waiting for it to appear randomly in CI (pytest documentation, "Flaky tests"). JUnit and TestNG offer comparable retry annotations in the Java ecosystem, and the same governance principle applies regardless of language: a retry mechanism is a triage tool, not a fix, and its use should be logged somewhere a human reviews.

# Illustrative example, not from a real codebase.
# A deliberate, bounded, logged quarantine — not a blanket suite-wide retry.

import pytest

@pytest.mark.flaky(reruns=2, reruns_delay=1)
@pytest.mark.quarantine(
    ticket="QA-4821",
    owner="checkout-team",
    reason="test-order dependency on shared inventory fixture",
    expires="2026-10-01",
)
def test_inventory_decrements_on_order_placed():
    ...

In the Java ecosystem, JUnit 5 and TestNG offer comparable bounded-retry mechanisms, typically implemented through a custom extension or a RetryAnalyzer interface rather than a built-in annotation in JUnit's core, which is itself a useful design signal: several major frameworks treat automatic retry as something a team should have to deliberately wire in and configure, rather than something that should be trivially available as a global default across an entire suite.

// Illustrative example, not from a real codebase.
// TestNG retry analyzer, applied deliberately to one test class,
// not globally to the whole suite.

public class BoundedRetry implements IRetryAnalyzer {
    private int attempts = 0;
    private static final int MAX_RETRIES = 2;

    @Override
    public boolean retry(ITestResult result) {
        if (attempts < MAX_RETRIES) {
            attempts++;
            return true;
        }
        return false;
    }
}

@Test(retryAnalyzer = BoundedRetry.class)
public void testOrderConfirmationEmailSent() {
    // Retry is scoped to this one test, with a hard cap,
    // and the retry count is visible in the test report —
    // not silently absorbed into a single pass/fail result.
}

CI-vendor-native detection. CircleCI's Test Insights feature automatically classifies a test as flaky when it both failed and passed on the identical commit within a 14-day rolling window, and surfaces per-test flakiness counts, success rates, and run times directly in its dashboard without requiring a separate tool (CircleCI documentation). GitHub Actions does not provide a first-party flaky-test classification feature at the platform level in the same way, but supports rerun-on-failure at the job or step level through workflow configuration, and third-party actions and community tooling have grown up specifically to add flaky-test tracking on top of Actions workflows. Buildkite similarly offers a Test Analytics product built specifically around tracking flaky and slow tests across a suite's history rather than treating each build in isolation. The practical distinction for engineering leaders evaluating CI-native features is whether the platform merely lets you retry (which any CI system supports) versus whether it tracks and classifies flakiness as a distinct, queryable signal over time (which fewer platforms do natively), and whether that classification is visible to the whole engineering organization by default or buried in a report nobody has a reason to open.

A related but separate decision is where retry configuration should live. Retry logic embedded directly in test code (a @Retry annotation on a specific test) keeps the decision close to the person with the most context, but scales poorly as an audit mechanism because there is no single place to see every retry-enabled test across the suite. Retry logic configured at the CI pipeline level (a global rerun-failed-jobs setting) is easy to audit but strips away the per-test judgment that makes retry a legitimate triage tool rather than a blanket suppression. The healthiest pattern observed across the tooling reviewed for this article combines the two: retry is enabled per test, in code, with a mandatory linked reason, but the CI platform or a lightweight custom script aggregates every retry-enabled test into one queryable list so the aggregate scope of the exception is always visible to whoever owns suite health.

Dedicated third-party flaky-test platforms. A category of specialized tooling has emerged specifically to detect, quarantine, and report on flaky tests across CI providers, generally by ingesting test result history and applying statistical classification beyond a single pass/fail-on-same-commit rule. These tools are worth evaluating for organizations with large suites (several thousand tests or more), high test parallelism, or multiple CI pipelines feeding into one release process, where the volume of test-result history makes manual tracking impractical.

The build-versus-buy decision should follow suite scale, not organizational size alone. A twenty-person startup with three hundred tests does not need a dedicated flaky-test platform; a spreadsheet, a pytest.mark.flaky convention with mandatory ticket linkage, and a weekly five-minute review is proportionate. An enterprise with fifty thousand tests across a dozen services and a dedicated release engineering function almost certainly benefits from CI-native or third-party classification tooling, because manual tracking at that scale simply will not happen consistently, and the cost of an undetected flakiness spike at that scale is proportionally larger.

How the Calculus Differs by Organizational Stage

Startups. Suite size is small enough that manual root-cause triage on every flaky failure is genuinely feasible, and the return on building formal governance infrastructure is low relative to the return on just fixing tests as they flake. The main risk at this stage is not under-investment in tooling; it is that early rerun habits, formed casually under deadline pressure with a five-person engineering team, calcify into unexamined norms that persist and worsen as the team and suite grow tenfold. The cheapest intervention at this stage is a lightweight rule: no reruns flag without a linked ticket, enforced by convention and code review rather than tooling.

Scale-ups. This is the stage where flakiness typically becomes visibly costly and where most of the framework in this article earns its keep. Suite size has grown past the point where any one engineer has full context on the whole suite, parallelization has usually been introduced to control CI runtime (which mechanically increases exposure to shared-state and resource-exhaustion flakiness), and the team is large enough that informal author-ownership starts breaking down. This is the natural point to introduce a formal flakiness rate metric, a quarantine policy with expiration, and an explicit ownership model — code-path ownership is usually the right fit here, ahead of a dedicated reliability function, which is rarely justified yet.

Enterprises. Suite scale, organizational complexity, and the number of teams contributing tests all argue for a dedicated test-reliability function and investment in CI-native or third-party flaky-test classification tooling. The governance challenge shifts from "do we have a process" to "is the process actually enforced across dozens of teams with varying levels of buy-in." At this scale, an unenforced flakiness budget is worse than no budget at all, because it creates the appearance of governance without the substance, which is its own form of the measurement-validity problem this article opened with — a metric that exists on a dashboard nobody acts on is not meaningfully different from no metric at all.

The transition between these stages is rarely announced by a company milestone; it is usually announced by the suite's own behavior. A useful signal that a startup has crossed into scale-up territory, from a testing standpoint, is the point where CI runtime pressure first forces a parallelization decision, because that decision is what mechanically exposes shared-state and ordering flakiness that had been dormant under sequential execution. A useful signal that a scale-up has crossed into enterprise territory is the point where code-path ownership stops being sufficient because a meaningful fraction of flaky tests span more than one team's owned code path, and resolving them requires someone with cross-team authority to arbitrate rather than two teams informally working it out. Engineering leaders do not need to wait for a headcount threshold to notice these signals; the suite itself will show them well before an org chart does, for any team paying attention to the metrics described earlier in this article.

Where This Framework Is Overkill

A reasonable question at this point is how a team tells the difference between "we are too small for this yet" and "we are already past the point where this would help and just haven't noticed." The honest answer is that the trust-audit checklist later in this article doubles as that test. A team that can already answer every item on it accurately, without needing to introduce a new dashboard or process to do so, does not need the formal apparatus described in this section, because the apparatus exists to produce answers a team cannot currently produce on its own. A team that cannot answer most of the checklist, regardless of its headcount or suite size, has already outgrown informal tracking whether or not it has outgrown its office space.

Not every test suite needs a flakiness budget, a dedicated ownership model, and quarantine-aging dashboards. A small suite (low hundreds of tests) maintained by a stable, small team that already investigates every red build as a matter of course does not need formal governance layered on top of good existing habits — the overhead of tracking metrics and running a budget review would exceed the problem it solves. Similarly, a test suite feeding a low-stakes internal tool, where a missed regression has limited blast radius and slow feedback loops are tolerable, does not warrant the same rigor as a suite gating a payments or healthcare release pipeline. The framework in this article scales in proportion to two variables: how large and parallel the suite has become, and how costly a false-negative release decision would be. Applying enterprise-grade flakiness governance to a ten-person startup's three-hundred-test suite is itself a form of the same mistake this article warns against — optimizing a measurement process instead of the underlying decision it is meant to support.

Practical Checklist: Auditing Your Own Test Suite's Trust

Use the following as a standalone diagnostic, independent of the triage checklist earlier in this article, to assess organizational health rather than a single test's root cause.

• Can anyone on the team state the current suite-wide flakiness rate, or would they have to guess?
• Is there a quarantine list, and does every entry on it have an owner, a reason, and an expiration date?
• Has any test been in quarantine for more than 90 days with no activity?
• Does your CI configuration apply reruns per-test with a documented reason, or suite-wide as a blanket setting?
• When a build goes red, is the team's default reaction to investigate or to rerun?
• Is there a mean-time-to-fix trend for flaky tests, and is it improving or degrading?
• Does anyone outside the immediate test author have visibility into which tests are flaky and why?
• Has a real production defect ever been traced back to a test that was dismissed as "just flaky"?

An organization answering "no" or "don't know" to more than two or three of these has a measurement-validity problem in its release pipeline, whether or not it currently feels like one.

Frequently Asked Questions

Is a flaky test always a defect in the test, never in the code under test? No, and conflating the two is one of the most consequential mistakes covered in this article. A test can be flaky because of a genuine intermittent defect in the system under test — a race condition, a timing-dependent data corruption bug — that only manifests under specific, non-reproducible conditions. The diagnostic checklist above exists specifically to force that distinction before any suppression mechanism gets applied.

Should flaky tests be deleted instead of fixed? Sometimes, and this is an underused option. If a test's value (the regression risk it actually protects against) is low relative to its maintenance cost and flake rate, deletion is a legitimate and often correct outcome, particularly for brittle end-to-end tests duplicating coverage that a more targeted integration or unit test already provides. The mistake is defaulting to indefinite quarantine instead of making an explicit fix-or-delete decision.

Does test parallelization cause flakiness, or just expose it? Almost always the latter. Parallelization rarely introduces new nondeterminism from nothing; it exposes shared-state and ordering assumptions that were already present but invisible under sequential execution. Teams that see a flakiness spike after introducing parallel test execution are usually looking at pre-existing test debt becoming visible, not a new class of bug created by parallelization itself.

How does a flakiness budget interact with a release freeze? It should be a defined trigger, not an ad hoc judgment call made under pressure. A workable pattern is: when the flakiness rate or quarantine list breaches its defined ceiling, new quarantine additions are frozen and the team allocates a fixed portion of its next sprint to working the backlog down before the ceiling resets, mirroring how an SRE error-budget breach typically triggers a reliability-focused work allocation rather than an indefinite halt to all other work.

Is a 0% flakiness rate a reasonable target? No. Given the sources of nondeterminism outlined in this article's taxonomy, particularly genuinely external dependencies and infrastructure-level resource variance, a literal zero is not a realistic target for most suites beyond a small size, and chasing it consumes effort disproportionate to the release-risk reduction it buys. The goal is a known, bounded, actively managed rate, not zero.

Who should decide the flakiness budget's ceiling, if there is no universal benchmark? The team accountable for release decisions, informed by the actual cost of both a false alarm and a missed regression for that specific pipeline. A pipeline gating a healthcare or payments release, where a missed regression is expensive and slow to detect in production, should set a tighter ceiling and invest more in fixing over quarantining than a pipeline gating an internal admin tool. The number matters less than the fact that someone with release authority set it deliberately, on those grounds, rather than inheriting whatever rate the suite happened to drift to.

Does adopting a flakiness budget mean release velocity slows down? Not durably, though it can slow down briefly during an initial backlog paydown, similar to the short-term cost of any technical debt remediation. The Spotify data point cited earlier is instructive here: their intervention was pure measurement and visibility, with no retry-policy change and no dedicated team pulled off feature work, and it still produced a meaningful reduction in flakiness within two months. Velocity typically improves once a team is no longer spending unmeasured, unacknowledged time on flaky-failure interruptions scattered across every engineer's week, which is a real cost even when nobody has put a number on it.

What is the single highest-leverage first step for a team that has never tracked flakiness at all? Instrument the suite to record, per test, whether each run passed on the first attempt or required a rerun, and make that data visible somewhere the whole engineering team can see it without asking for it — a dashboard, a weekly digest, a channel post, the specific mechanism matters less than the visibility. This is deliberately the lowest-effort recommendation in this article, and it is listed last precisely because it is the one most teams can start this week without waiting for a budget, a tooling purchase, or an organizational redesign. The Spotify result cited throughout this article is worth restating in this specific context: visibility alone, with no other change, cut their measured flakiness rate by a third in two months. Most of the governance apparatus described elsewhere in this article exists to sustain and scale that same effect once a team has outgrown what visibility alone can accomplish, not to replace it as the starting point.

The QAtronic Perspective

Most engagements where QAtronic reviews an existing test automation suite surface this same pattern: substantial test coverage exists, CI is fully configured, and the team still cannot fully trust their own green builds. When that is the underlying issue, the fix is rarely more tests. It is a structured audit of the existing suite's reliability — classifying which failures are genuine, which are test-infrastructure artifacts, and which are unmocked or under-isolated dependencies — paired with a concrete ownership and quarantine-governance model the team can actually sustain after the engagement ends. If your organization suspects its CI signal has quietly stopped meaning what it used to, that audit is a more useful starting point than adding more tests on top of an already unreliable foundation.

The Decision This Article Is Actually About

Every engineering organization running automated tests already has a de facto answer to the question "what does our test suite's green checkmark certify?" Few have ever stated that answer explicitly, and fewer still have checked whether the answer they would give if asked matches the answer their CI configuration actually implements. The gap between those two answers is where flaky tests do their real damage — not in the individual failed run, but in the accumulated, undocumented redefinition of what passing means.

The concrete decision for an engineering leader reading this is not whether to eliminate flakiness entirely, which is neither realistic nor worth the cost. It is whether your organization can currently produce a straight answer to a simple question: if a test fails once and passes on rerun, does anyone know why, and is that reason recorded anywhere a future engineer will find it? If the honest answer is no, the test suite has already stopped functioning as a measurement instrument for release risk, regardless of how green the dashboard looks this week.