Flaky Test Root Causes in Non-Reproducible Environments

Flaky tests aren't rare. A 2025 empirical study found that roughly half of software practitioners encounter them daily, weekly, or monthly. This isn't an edge case afflicting teams with poor test hygiene. It's a pervasive condition of modern software development, and its costs are documented precisely enough that continued hand-waving is indefensible.

Atlassian has put a number on it — more than one hundred thousand developer hours lost per year to flakiness investigation. Those hours belong to capable engineers chasing failures that don't reproduce reliably, that resist bisection, that vanish when scrutinized. At Slack, before automated flakiness detection existed, the majority of failing builds were caused by unstable tests rather than actual regressions. Google's research into CI failures found the same pattern at scale. Most failures that surface as apparent regressions trace back to test instability, not defects in the code under test. These are organizations with serious investment in test infrastructure, and they still spent years chasing ghosts.

The proportion of teams experiencing flaky tests has grown substantially in recent years, tracking the rise of distributed systems and heterogeneous CI infrastructure. As test suites expand, as pipelines span multiple operating systems and architectures, and as dependency graphs deepen, the surface area for environmental variance increases. The problem scales with the sophistication of the system. That's precisely backwards from what engineering maturity should produce.

The most insidious dimension of this cost is its invisibility. A developer who spends two hours determining that a failing build wasn't a real regression doesn't log that time against a test reliability ticket. It vanishes into ambient overhead, distributed across enough individuals that no single line item surfaces in a budget review. The cost is real, it's large, and it's almost perfectly structured to avoid organizational scrutiny.

The Standard Taxonomy of Flaky Test Root Causes and Where It Stops Short

The canonical list of flaky test causes is well established. Asynchronous wait issues, including fixed sleep() calls that assume a particular execution speed, represent the largest single category in large-scale benchmark data, accounting for a plurality of documented cases. Concurrency and race conditions constitute a significant secondary category. Test order dependency and shared mutable state appear consistently across postmortems. DOM and selector fragility accounts for a smaller share than timing and data issues combined, per analysis from QA Wolf. External API and third-party instability rounds out the standard list.

Each of these categories is real, and each has legitimate remediation strategies. The taxonomy carries an implicit assumption that almost no one names directly, though — the environment is a constant. Same OS, same runtime, same dependency versions, same available resources, run after run, machine after machine. That assumption fails in virtually every modern software organization, and when it fails, fixing the test code produces only temporary relief.

A 2025 peer-reviewed study by Parry and colleagues examined hundreds of flaky tests across thousands of suite runs and found something worth sitting with. Flakiness failures cluster. Multiple tests fail simultaneously, sharing the same root cause at the same moment. That pattern directly contradicts the standard model, in which each test fails independently due to its own internal defect. Clustered failure is the signature of an upstream trigger acting on the test suite as a whole rather than on individual tests in isolation. That upstream trigger is almost always environmental. When three unrelated tests fail together after a CI dependency cache expires, the environment is the common cause. The standard taxonomy has no category for it, which means teams applying that taxonomy will diagnose the wrong thing every time.

How Resource Variance and Timing Turn Environment Differences into Test Failures

A 2024 IEEE study found that nearly half of flaky tests are resource-affected, meaning they pass or fail depending on CPU, memory, or computational resource availability at execution time.

Consider what that looks like in practice. A test that passes consistently on a developer's laptop with spare cores will fail reliably on a loaded CI runner executing identical code. The test hasn't changed. The application hasn't changed. The failure is a product of the execution context. A timeout value that's generous under low load becomes insufficient when the container is throttled. An async wait that works reliably at low utilization produces a race condition under resource pressure, because the process that should complete before the assertion simply doesn't finish in time. The test looks wrong. It isn't.

Screen resolution, browser version, and available system resources also vary between local and CI environments in ways that produce rendering and timing failures that resist reproduction without controlling the environment. A developer running a headed browser on a high-resolution display isn't running the same test as a CI agent running headless in a throttled container. The inputs differ materially even when the test code is identical.

Async timing fixes, better explicit waits, and retry logic reduce the frequency of failure under constrained resources; they don't eliminate the underlying variance. That variance reasserts itself every time CI infrastructure changes, every time a new runner type is provisioned, every time a build queue lengthens and containers are allocated under greater load. Treating resource variance as a test-code problem is treating a symptom as a cause. The symptom returns. It always does.

Dependency Drift as a Source of Flakiness That Accumulates Silently Over Time

Nobody on your team decided to change the dependency. It just changed.

A requirements.txt with floating version ranges resolves to whatever the current index serves at installation time. A container image tagged :latest pulls whatever the registry currently hosts. Dependency update automation silently bumps transitive packages between runs. CI caches that expire force fresh resolution while local caches hold stale versions from weeks ago. Each of these events is small, undocumented, and effectively invisible unless you're specifically instrumented to detect it.

The Docker evidence sharpens this point considerably. A study of thousands of Docker builds found that only a small fraction of rebuilt images matched the original installed package versions exactly; bitwise identity between builds was virtually never achieved (Malka et al., 2026). Even in an ecosystem specifically designed for reproducibility, rebuilding from the same Dockerfile at different points in time produces a materially different artifact in the overwhelming majority of cases. Docker narrows the variance. It doesn't eliminate it.

What this produces in a test suite is a failure mode that looks like intermittent instability but is actually deterministic environmental change. A test that was green against library version N becomes flaky against version N+1 with no modification to the test or the application code. The failure appears random because no one consciously changed anything. But the library changed, the environment changed, and the test is surfacing a real behavioral difference. Because the dependency drift is invisible, the test gets blamed.

Research examining reproducibility failures in software engineering artifacts found that omitting machine-readable dependency lists or container specifications causes a meaningful fraction of reproducibility failures even when practitioners are actively trying to replicate prior work (Siddiq et al., 2025). If deliberate reproducibility attempts fail at this rate due to documentation gaps, the accidental reproducibility that most CI pipelines rely on is far more fragile than it appears.

OS and Architecture Differences That Make a Test Correct on One Platform and Broken on Another

Cross-platform flakiness has a different structure than timing or state issues, and that structural difference shapes how you diagnose it. A timing-related flaky test fails intermittently on the same platform. A cross-platform failure fails deterministically on one target and passes deterministically on another. It reads as flakiness only because teams run tests across multiple environments and aggregate results without adequate metadata about which environment produced which result.

The sources are well understood. File path separators and line-ending conventions produce failures that are invisible on case-insensitive macOS file systems and consistent on the case-sensitive Linux systems that run CI. ARM versus x86 differences introduce floating-point precision disparities, instruction ordering effects, and binary dependencies compiled for one architecture that don't behave identically on the other. System library versions differ across OS distributions; a test linking against a system-level TLS library behaves differently on one Ubuntu LTS release than the next, even when application code is identical.

As development teams standardize on Apple Silicon laptops locally and deploy to x86 Linux in CI and production, this failure category is actively growing. The failure mode is particularly insidious because the test is stable for every developer individually. Each developer, on their own machine, sees green. The test is reliably broken in CI. Because no single developer observes intermittency on their own platform, the signal reads as a CI infrastructure problem rather than a cross-platform test problem. Both descriptions are technically accurate. The actionable root cause is the uncontrolled platform difference.

I've watched skilled engineers who know their stacks cold lose hours to this, not because they lacked the knowledge to solve it, but because the environment they were debugging against wasn't the environment that produced the failure. You can't diagnose what you can't see. That's a tooling gap, not a knowledge gap, and conflating the two is how teams justify underinvesting in the infrastructure.

Why Rerunning a Failing Test Is a Symptom Response, Not a Diagnosis

The operational playbook for a flaky test is familiar: mark it flaky, add retry logic, quarantine it from blocking the build, assign it for later investigation. When the root cause genuinely lives in the test code, this is defensible. Repair the bad selector, add a proper wait, isolate the shared state. Retry and quarantine buy time for a real fix.

The approach fails for environment-driven flakiness, and the failure mode is self-reinforcing. Retrying the same test in the same environment encounters the same resource constraints or the same dependency version skew that caused the initial failure. The retry logic masks the failure rate from dashboards, which reduces the apparent urgency of investigation, which delays the root-cause work that would actually resolve it. The quarantine list grows. Teams develop a graveyard of tests that don't block CI but also don't catch regressions. Slack's pre-automation state, where only a minority of builds were passing, illustrates the endpoint of this trajectory — when the false-failure rate is high enough, the entire test suite loses its signal value, and teams begin discounting results as a matter of routine.

The deeper problem is diagnostic. Retry counts tell you a test is flaky. They don't distinguish between a race condition in the test code, a dependency version that changed between the local run and the CI run, and a resource constraint that caused an async wait to time out. All three failures present identically in most CI dashboards — a red build, a retry, an eventual green. The information needed to differentiate them (specifically which runtime version, which library versions, and which system resources were present during the failing run) isn't captured by default in most CI systems.

Without knowing what was actually running when a test failed, every investigation starts from reconstruction rather than from evidence. Reconstruction almost always ends the same way: "we can't reproduce it." The environment that produced the failure no longer exists and was never recorded.

What It Takes to Make a Test Environment Actually Reproducible

Reproducibility is a precise term. It doesn't mean "everyone is roughly on the same OS." It means bit-for-bit identical dependency resolution across every context where tests run — developer laptop, CI runner, staging environment, production deployment. Any divergence from that standard creates a surface where environment-driven flakiness can take hold.

The requirements are well defined. Every package must be resolved to an exact content-addressed hash, not a version range. Lockfiles must travel with the code and must be enforced in CI, not treated as advisory artifacts. Runtime and toolchain versions must be pinned in the environment definition itself, not assumed from whatever the host system happens to provide. Environment definitions must be machine-readable and version-controlled, so that a new developer or a new CI node activates the same environment from a single command rather than following a prose setup guide of uncertain vintage.

The practical test for reproducibility is concrete: can you check out a commit from six months ago and reproduce the exact environment it was tested in? If the answer is no, then a failure observed today can't be reliably attributed to a regression versus an environmental change. That determination is the entire point of a test suite.

Content-addressed environment management, where every developer, CI runner, and deployment target resolves the same dependency closure from the same inputs, addresses this directly. The environment becomes a versioned artifact that can be checked out, reproduced, and audited the same way application code is. One organization compressed environment setup time from two weeks to under two hours through standardization and automation along these lines. That result shouldn't be surprising. Eliminating environment-driven flakiness and eliminating onboarding friction are not two problems with two solutions. They're the same problem.

Environment Reproducibility as Part of a Broader Supply Chain Integrity Posture

A flaky test caused by a silently updated transitive dependency is the benign version of a supply chain event. The dependency changed, the behavior changed, no one noticed until CI started turning red intermittently. Expensive, but recoverable.

The malicious version of the same mechanism isn't hypothetical. The 2025 compromise of the tj-actions/changed-files GitHub Actions workflow is the canonical recent example — attackers modified version tags so that workflows referencing those tags executed a malicious commit, potentially affecting tens of thousands of repositories. The attack vector was precisely the mutability that causes dependency drift in test suites. A tag that should have been immutable was not. Workflows that pinned to a tag rather than a content-addressed commit hash were exposed; those that pinned by hash were not. The difference between a reliability problem and a security incident was a single character in the configuration.

Software Bills of Materials provide an inventory baseline, an exact record of every dependency present in a build, enabling rapid exposure assessment when a vulnerability is disclosed. A compliance-snapshot SBOM generated at the end of a build and filed away doesn't help with drift. An SBOM from a prior release doesn't represent the software running in today's build if dependencies are unpinned and resolving differently each time.

A content-addressed, pinned environment produces an SBOM that's accurate and current because the environment it describes is the environment that actually ran. Provenance becomes a byproduct of reproducibility rather than a separate audit exercise. Software provenance and SBOM requirements have moved from best practice to mandate under U.S. Executive Order 14028 and the EU Cyber Resilience Act. Teams that solve environment reproducibility for test reliability reasons acquire compliance posture as a consequence. The work is the same work.

How Platform Engineering Teams Can Eliminate Environment-Driven Flakiness at the Infrastructure Level

Environment-driven flakiness is an infrastructure problem. The fix belongs to the platform engineering team, not to individual developers triaging test-by-test root causes. Assigning environment-driven failures to application developers is the organizational equivalent of asking each driver to repave their section of road. The road has an owner. So does the environment.

The platform engineering mandate here is specific — provide developers with base environments that are reproducible by construction. Not "here is a Dockerfile, follow the wiki," but versioned, activatable environments consistent across every surface where code executes. When the local environment and the CI environment are the same artifact activated from the same definition, the "works locally, fails in CI" failure class becomes structurally impossible rather than merely improbable.

The leverage points are concrete. Enforce lockfile discipline in CI — fail the build if the lockfile is out of sync with the manifest, before any test runs. Pin runtimes and toolchains in the environment definition itself, not in a README that developers may or may not follow. Instrument CI to capture environment metadata on every run (specifically runtime versions, resolved dependency hashes, and available resources) so that a flaky test failure can be correlated with an environmental change rather than requiring a developer to reconstruct what was running from memory. Make that metadata queryable across historical runs. A pattern like "this test started failing after the CI runner was upgraded" should be visible from the data, not from tribal knowledge.

When the development environment is treated as a versioned artifact with a defined interface, a managed upgrade path, and owned reliability, several things improve at once: environment-driven flakiness recedes, onboarding compresses, SBOM generation becomes accurate and continuous, and CI pipelines acquire a stable foundation. These aren't separate returns on separate investments. They follow from one thing being done correctly, and the platform team is the group positioned to do it.