Eliminating Works-on-My-Machine Failures in CI Pipelines
Drift isn't a catastrophic event. It's the accumulated residue of individually harmless decisions made over months by people trying to ship.
On the developer side, the accumulation is almost entirely invisible. A developer installs a tool globally because it's faster than adding it to the project manifest. Another upgrades their Node runtime for a side project and never checks whether it affected this one. A third adds a shell alias to work around a build quirk specific to their laptop. None of these actions feel significant in the moment, and none appear in any configuration file the team shares. Three months later, you have six developers running six subtly different environments, and nobody can reconstruct when or how the divergence started, because there was no incident to point to.
On the CI side, drift is often structural rather than behavioral. Floating image tags like node:20 silently resolve to a different underlying image each time the runner spins up. Pipeline dependencies are fetched fresh from external registries at build time with no version pins. CI providers update their base runner images on schedules that teams rarely track and almost never audit. The environment shifts, and the pipeline keeps producing green checkmarks right up until it doesn't.
The layered configuration problem compounds both sides. Environment variables live simultaneously in .env files, cloud secret managers, and pipeline settings, each layer capable of overriding the others in ways that only become visible when something breaks. A required variable added to staging config and to local .env files but omitted from the pipeline definition is the canonical example: the feature works everywhere a human touches it and fails silently in CI or production. Nobody wrote that variable down as a dependency. It just existed, on the machines where it existed.
What makes this particularly expensive is that drift produces no alarm. When a developer's local runtime version diverges from the one the pipeline assumes, no warning fires. The gap widens quietly, for months, until something fails in a context where the person who would recognize the problem isn't present.
What it costs to treat this as a one-off debugging problem
The instinct is understandable: re-run the pipeline, pull in a senior engineer, add a workaround, and move on. These responses address the symptom while leaving the structural gap entirely open, ready to surface again two sprints later in a slightly different form.
Each environment archaeology session costs real hours. The work involves reconstructing CI conditions locally, comparing tool versions across two or more systems, and hunting undocumented assumptions that were never written down because the person who made them considered them obvious. The frustration, after enough of these sessions, isn't that the problem is hard. It's that it didn't need to exist.
Onboarding is where accumulated cost becomes undeniable. A new developer sets up a fresh machine against a README last updated months ago, and every gap between the documented state and the actual environment costs hours before they write a single line of product code. Onboarding designed to take a day routinely stretches to three or four when environment assumptions aren't codified. Time-to-first-commit is a direct, measurable proxy for undocumented drift.
Flaky tests are a particularly corrosive dimension of this cost. When environment inconsistency produces intermittent CI failures, tests that pass locally and fail in the pipeline without a clear pattern, teams begin ignoring pipeline results and re-running jobs hoping for green. This is precisely the behavior that defeats continuous integration. A CI system engineers have learned to distrust isn't a safety net; it's overhead with a false sense of security attached.
The right framing is a velocity tax. Configuration drift isn't a one-time bug. It's a recurring drag on every deployment, every onboarding, every incident response, paid continuously until the environment definition is treated as code.
Why documentation and convention don't hold the environment together
The reflexive response to "works on my machine" failures is documentation: a longer setup guide, a pinned Slack message, a more detailed onboarding checklist. This response is well-intentioned and structurally inadequate, and more effort invested in it doesn't change that.
Documentation describes an intended state. It has no mechanism to enforce that state. A README can't detect when a developer's local tool version has diverged from what it specifies. A setup checklist executed by ten different developers in ten different moments produces ten subtly different environments, because manual steps are interpreted differently depending on who follows them, when, and with what prior context. The person who wrote the checklist and the new hire following it six months later aren't operating with the same assumptions. They never are.
Convention fails for the same reason, only more slowly. A team norm that says "always use Node 20" produces as many Node versions as there are developers, because there's no enforcement mechanism in the development environment itself. Norms are maintained by social pressure and attention, both of which degrade under the normal pressures of a shipping product team. The convention holds until someone is under deadline pressure and takes a shortcut. That shortcut is harmless, right up until it isn't.
The CI parity problem is particularly acute here. Even teams that document their local setup carefully often never document the CI environment's exact state, because CI configuration lives in a separate YAML file maintained by different people, and nobody cross-references the two systematically. The two documents drift apart because they were never actually connected. Nobody finds out until something is on fire.
Parity between local and CI environments cannot be maintained by human discipline alone. Both environments must be defined from the same source of truth: a file that lives in the repository, is versioned like any other code, and is machine-enforced rather than norm-enforced. Everything else is process without teeth.
What environment-as-code means in practice and what tools provide it
The definition is precise: the environment, including runtime versions, tool versions, system dependencies, and environment variables, is declared in a versioned file that lives in the repository alongside the application code. Any machine that checks out that repository and runs the designated setup command gets an identical environment. The definition is reviewable, diffable, and auditable. This discipline was applied to application infrastructure years ago; it's now arriving at the development environment itself.
Several mature tools make this concrete, and the differences between them matter.
Nix and Nix flakes treat packages as pure functions of their inputs: the same inputs always produce the same outputs, down to binary content. This is the deepest reproducibility guarantee available in the ecosystem. The learning investment is real, but for teams with complex dependency graphs or strict audit requirements, no other tool matches it.
Devbox exposes a more accessible surface over that same foundation. A devbox.json file declares the tools and runtimes the project needs; a devbox.lock file resolves those declarations to exact versions. Any developer running the shell command gets bit-for-bit identical tool versions. Devbox uses Nix under the hood, capturing its reproducibility guarantees without requiring fluency in the Nix language, and it works across macOS, including Apple Silicon, Linux, and WSL.
Flox extends the Nix foundation toward the full software lifecycle: local development, CI, and deployment contexts managed from a single environment definition. Software Bill of Materials generation and provenance tracking are built into the environment definition rather than bolted on after the fact, and single-command onboarding is treated as a first-class product requirement.
Dev Containers, the Microsoft-originated specification supported natively by VS Code and GitHub Codespaces, defines the environment inside a Docker container via a .devcontainer/devcontainer.json file. The reproducibility guarantee here depends entirely on whether the underlying Dockerfile pins to a specific image digest or a floating tag. Floating tags are the common failure mode and reintroduce exactly the kind of drift the approach is meant to eliminate.
mise is a polyglot runtime version manager lighter-weight than the Nix-based options. For teams whose primary need is runtime version consistency rather than full environment isolation, it's a practical and well-supported choice that fits into existing workflows without significant disruption.
The key differentiator across these tools is whether they pin transitive dependencies and system libraries or only the top-level runtime. Tools that pin only the top level leave a class of drift open. Teams operating across mixed macOS and Linux CI environments should verify that a tool's lock file resolves consistently across both architectures; Nix-based tools have the strongest story here.
The selection criterion that matters most isn't which tool is theoretically optimal. It's which tool the team will actually invoke in CI as well as locally. A tool used only locally solves nothing.
The rule that makes local-CI parity mechanical rather than aspirational
The rule is simple: whichever environment tool a team adopts, it must be invoked identically in the CI pipeline. Not approximated, not re-implemented in pipeline-native YAML, not "kept in sync" manually. Simple enough to write on an index card, and frequently violated in practice.
Once the same environment definition file drives both local setup and CI setup, parity is structurally enforced. It can't drift unless someone changes the definition file, which is a visible, reviewable event. Drift becomes deliberate rather than accidental, and deliberate drift can be managed.
The implementation is concrete. For Devbox, the official GitHub Actions integration runs the same devbox.lock on the CI runner that resolves on a developer's laptop. For Flox, the same environment activated locally is activated in CI, pulled from version-controlled definitions. For Dev Containers, the CI job must build from the same devcontainer.json and the same pinned image digest, not a parallel Dockerfile maintained separately by a different team member who may or may not be keeping it current.
The failure mode to avoid is teams that define the local environment in one tool and then recreate what they believe to be the same environment in CI using native pipeline YAML steps. These two definitions diverge immediately and continuously. It happens because the CI configuration was inherited from a previous setup that nobody fully understood and nobody wanted to touch; now the gap between the two is precisely the problem the team set out to solve, reinstated by the implementation.
A complementary discipline is the build-once, promote-through-environments pattern: build the artifact once in the verified environment, then promote that artifact through staging and production rather than rebuilding at each stage. Rebuilding reintroduces the possibility that the environment shifted between stages.
One practical concern teams raise consistently is caching. Nix-based tools can cache the package store in CI, using GitHub Actions Cache, for example, so that environment setup is near-instantaneous without compromising what's being installed. Caching an unpinned environment undermines reproducibility. Caching a pinned one is just good engineering.
How a reproducible environment definition also solves the supply chain audit problem
Sonatype identified over 454,600 new malicious packages in 2025 alone, bringing the cumulative total past 1.2 million. A CI environment that fetches dependencies without pinning is exposed to the full surface of that problem on every build. Most teams are aware of this in the abstract and have done relatively little about it in practice.
A pinned, reproducible environment knows exactly what it contains: tool versions, library versions, and cryptographic hashes. This is the prerequisite for any meaningful security audit, and it's the prerequisite most teams currently lack. You can't audit what you can't enumerate.
Software Bills of Materials are machine-readable inventories of every component, version, and origin in a build. When an environment is reproducible and content-addressed, generating one is a byproduct of the build process rather than a separate forensic exercise conducted after something goes wrong. Most SBOMs generated today are compliance snapshots produced at the end of a build and filed without being acted on. A live, accurate inventory tied to the environment definition is a fundamentally different artifact.
Nix's content-addressed store computes hashes for all dependencies, making tampering harder to conceal. The lock file fixes both versions and source hashes, which reduces the window for dependency substitution attacks, the kind where a legitimate package is swapped for a malicious one between build invocations.
Provenance attestations, including SLSA and in-toto frameworks, link each build artifact back to the exact source commit and build environment. These attestations are only meaningful when the build environment itself is defined and stable. An attestation pointing at a floating environment proves very little.
The regulatory context is accelerating this discipline. U.S. Executive Order 14028, the EU Cyber Resilience Act, and sector-specific compliance frameworks increasingly require SBOM production and supply chain traceability. Teams that have treated environment definition as code will find compliance tractable. Teams that haven't will find it forensic, expensive, and retroactive.
The CI environment itself is an attack surface, not merely a conduit for building software. A 2025 incident in which attackers modified historical Git tags of a widely-used GitHub Action exfiltrated CI/CD secrets from an estimated tens of thousands of repositories. Pinning action versions and verifying hashes applies the same reproducibility discipline to pipeline tooling that environment-as-code applies to the rest of the stack. The principle doesn't change because the artifact is a GitHub Action rather than an application dependency.
What changes for teams adopting this approach — onboarding, debugging, and day-to-day workflow
Onboarding is the clearest test of whether an environment is actually defined as code, because a new developer has no tribal knowledge to compensate for gaps. When the environment is properly defined, the sequence is git clone, one command to activate the environment, then work. No README gap to bridge, no undocumented workaround to stumble into, no hour lost to a version mismatch that whoever set up the machine three years ago would have recognized immediately.
Without this, onboarding stretches across multiple days as new developers encounter the accumulated distance between the documented state and the actual environment. That cost isn't dramatic enough to cause alarm on any given day, which is precisely why it never gets fixed.
Platform teams see the most immediate operational change. They stop fielding "my environment is broken" escalations. They stop maintaining parallel documentation for local and CI setup. They stop manually reconciling environment differences when a CI failure can't be reproduced locally. What they do instead is maintain one environment definition file that is the authoritative source for local, CI, and where applicable, deployment contexts. For platform engineers who've spent years playing environmental detective, that consolidation matters.
The debugging workflow changes in a way that's difficult to appreciate until you've experienced it directly. When a CI failure occurs, a developer activates the exact same environment locally and reproduces the failure deterministically. The debugging loop is honest because the environments are identical. There's no longer a category of failure that exists in CI but cannot be reproduced locally due to environmental difference. That category, in practice, accounts for a meaningful fraction of the hours spent on CI failure investigations.
One emerging stakeholder deserves explicit consideration: automated coding agents. Agents running in CI or in agentic development loops need the same environment rigor as human developers, more so, because they won't notice that something is subtly wrong and ask a colleague. An agent that provisions its own dependencies outside the pinned environment definition reintroduces drift through a non-human vector. The environment definition must cover agent execution contexts as deliberately as it covers developer workstations.
The organizational pattern that keeps environments from drifting back
Adopting an environment definition tool is necessary but not sufficient. Without an organizational pattern to maintain it, the definition file becomes stale within months and teams revert to familiar habits. This isn't a failure of character; it's a predictable outcome of incentive structures that reward shipping over maintenance.
The pattern that holds is treating the environment definition file with the same review discipline applied to application code. Changes to the environment definition require a pull request, a review, and a passing CI run against the new definition before they're merged. Environment changes become visible and intentional. Invisible drift cannot be managed; deliberate drift can.
Ownership matters more than tooling choice. A designated platform or developer experience function, however small, holds responsibility for the environment definition file across projects. This doesn't mean a single team controls all environments; it means someone is accountable for keeping the definition current and for reviewing changes that touch it. Without designated ownership, the file drifts toward neglect, because neglect is the path of least resistance when nobody's name is on the line.
Automated drift detection completes the pattern. CI jobs that periodically validate the environment definition against the actual runner state, or tools that alert when a lock file hasn't been updated in a configurable period, catch decay before it produces a failure. Environment debt is far cheaper to address before it becomes an incident.
When these practices hold, "works on my machine" loses its meaning as a complaint, because every machine, including every CI runner, resolves to the same environment. That happens when the environment definition is treated as infrastructure code: reviewed, versioned, and maintained with the same seriousness applied to the code it exists to support.
