Hermetic Builds and Why Non-Hermetic Builds Break CI Reliability

Whatever the build touches counts as risk: host tools, ambient environment variables, whatever the network happens to be serving that morning. A build is hermetic when it depends on nothing but its own declared inputs: no host-installed tools, no scavenged system libraries, no network calls slipped in mid-build. Reproducibility is the payoff. Same inputs, same build, same output, byte for byte, whether it runs on a laptop on a Tuesday afternoon or on a CI runner six months later. A non-hermetic build can hum along for months and then quietly stop working the moment something on the host changes underneath it, and that's the part that catches teams off guard every time. Bazel's own documentation names four usual suspects: arbitrary processing in build-script fragments, actions or tooling that create files non-deterministically, unversioned tools in system paths, and environment variables set by whoever happened to run the build. The host is never a neutral backdrop. Any build that borrows from it inherits that variability without ever admitting it did.

How ambient host state silently infiltrates a build

Non-hermetic dependencies don't show up in a lock file. They don't show up in a manifest either. Nobody catches them in code review, because there's nothing there to review, until the host changes underneath a team and the build breaks for reasons nobody can name on the first try, or the second, or sometimes the third.

Take a build script that calls python3 or make. It gets whatever binary the OS happens to provide, which is a different binary on macOS 14 than on Ubuntu 22.04, and different again once a CI runner image gets patched next Tuesday. Or a dependency gets pinned with a floating range, something like ^4.18.0, which resolves to one artifact the day it's written and a different one six months later once a patch lands upstream. Without a lock file, "compatible" is a moving target. Some builds read PATH, or CC, or JAVA_HOME, or a credentials variable sitting quietly in someone's shell profile, none of which exist in the same form on a CI agent. Others reach out to the network mid-build to fetch a dependency live, tying the output to whatever a registry happened to be serving at that exact second.

Local machines make this worse. They accumulate cached packages, globally installed tools, background daemons, leftover state from three unrelated projects, none of it appearing anywhere in the current project's declared dependencies. CI agents are cleaner by comparison, built fresh every run. I've long suspected that gap, between a cluttered laptop and a fresh runner, is the actual structural source of the oldest complaint in the industry: it worked on my machine.

There's a quieter failure mode too, one that took me embarrassingly long to recognize the first time I hit it: a build writes files back into its own source tree as a side effect. At that point the output depends on whether a previous build already ran on that exact machine. The build has started mutating its own inputs without telling anyone.

The failure modes that show up in CI when builds are non-hermetic

The numbers back this up. Not precisely, but directionally, which is the point. One study of CI failures found that non-reproducible failures, the kind that vanish cleanly on rerun in a fresh environment and trace back to transient infrastructure issues like disk corruption, network timeouts, clock skew, or container cold starts, made up a bit over a sixth of the sample. Another small slice got attributed directly to environment impact. Add those together and you've got a meaningful chunk of CI failures that have nothing to do with the code someone just wrote.

What actually costs teams time isn't how often this happens. It's how long it takes to figure out what happened. At the moment a build fails, nobody yet knows whether it's a flaky network call, a registry that shifted underneath them, a toolchain silently updated on the runner image overnight, or a genuinely broken container. The investigation ends up covering the whole host environment, not just the diff in the pull request. A transitive dependency can ship a breaking change with nothing in the current commit touching that library at all, invisible in review, because review only ever shows the code someone actually wrote.

Longer-horizon data tells a similar story. A study of failing CI runs in embedded open-source software found reconstruction failures driven mainly by changes in external dependencies (repositories getting removed, proprietary toolchains shifting underfoot, minor environment drift accumulating quietly) rather than mistakes in the build logic itself. Compilation failures in particular tend to trace back to mismatched dependencies and implicit environment assumptions baked in so deep they're hard to reproduce even locally.

Here's the asymmetry that matters, and it's the one I'd put money on every time. In a non-hermetic pipeline, a failure opens an investigation across host state, network conditions, registry behavior, and the entire dependency graph. In a hermetic pipeline, the failure scope stops at the declared source and its explicit inputs. Twenty minutes of triage versus most of a day, and anyone who's lived through both knows which one they want to be doing at 6pm on a Friday.

Why "just use containers" doesn't fully solve the problem

Containers shrink the exposure. They don't close it, and this is where a lot of teams stop thinking too soon. The base image is itself an external dependency that mutates over time, quietly, without changing its name. Pulling ubuntu:latest today does not get you what pulling ubuntu:latest got you six months ago, even though the tag never moved at all.

Build steps that reach out to package registries at runtime bring the exact problem containers were supposed to solve back in through the side door. A pip install or apt-get running inside a container isn't hermetic just because it's inside a container; it's still resolving against whatever the outside world happens to be serving right now. Build-in-Docker, where a Bazel action runs inside a predefined container supplying the binaries it needs, is a real improvement over nothing. It only closes the loop, though, if that image is version-pinned and rebuilt from a known snapshot, not floating on a mutable tag someone forgot about years ago.

There's a practical cost too, and it shows up on real hardware rather than in a diagram. Engineers on lower-spec laptops feel Docker's memory overhead directly, somewhere around 1.5 to 2 GB per running container, which matters when the whole point of the exercise was getting local and CI to actually match instead of diverging over resource limits.

The deeper issue is conceptual, though. Containers give you isolation. Hermeticity needs isolation plus input-pinning: knowing exactly what went into the build, not just that the outside world got partially fenced off while it ran. Getting there needs a build system where declaring inputs and sandboxing are structural, not something a team has to remember to do right every single time. Flox, a single-manifest environment manager built on Nix, is one approach to keeping those declared inputs consistent across laptops and CI from the start.

Bazel's approach to hermeticity: sandboxing build actions and controlling the toolchain

Bazel runs every build action inside a sandbox that can see only what's explicitly declared as an input for that action. An undeclared file gets no access, full stop. That's a genuinely different failure mode from the usual one: non-hermetic access fails loudly at build time, instead of succeeding quietly and leaving a landmine for whoever runs the build next.

Bazel also supplies its own toolchains (compilers, linkers, interpreters) rather than inheriting whatever happens to live in /usr/bin on the machine doing the building. That shuts the door on the unversioned-system-binary problem directly. And because every input to every action is fully declared, Bazel can cache actions individually and skip rerunning them when nothing relevant changed. It's a cache actually worth trusting, because hermeticity guarantees there's no invisible input hiding underneath it.

The scale numbers, where teams have published them, are hard to argue with. One migration of a large Python monorepo to Bazel with hermetic rules brought full builds down from something like 3 hours to 12 minutes, got incremental builds under a couple seconds at a cache hit rate north of 90 percent, and cut CI time by roughly six or seven times over, all while landing binary-identical outputs across runs. Bazel fits monorepos and multi-language codebases especially well; it handles C, C++, Rust, Python, and others, and scales from small repos up to very large ones.

None of it comes free, and I want to be honest about that instead of glossing over it. Writing BUILD files is real, ongoing work, and the learning curve is steep enough that Bazel is a poor fit for a small, single-language project that never needed monorepo-scale caching in the first place. Bring in a hammer this heavy for a birdhouse and the team spends more time fighting the tool than the problem it was meant to solve.

Nix's approach: treating the entire environment as a pinned, reproducible program

Nix comes at the same problem from a different direction. Instead of sandboxing individual actions, it treats the whole environment, toolchains, libraries, shell utilities, environment variables, as a pure functional program. Every package builds from an exact, cryptographically hashed description of its own inputs, so two systems building the same package from the same description land on the same result.

The modern version of this is Nix Flakes. A flake is a hermetically sealed description of inputs and outputs, whether that's a dev environment, a container image, or a deployment config, pinned by a flake.lock file recording the exact commit hash of every dependency involved. What that closes, in a way containers alone don't, is the environment itself becoming a versioned artifact sitting in the repo next to the code. A developer's shell, the CI runner, and production can all activate from the same lock file, which turns drift between those three contexts into something you notice on sight rather than a mystery bug surfacing three weeks later, usually on a Friday, usually right before a release.

There's a real payoff around onboarding too. Teams using Nix Flakes have reported cutting new developer setup from roughly two days down to twenty minutes, because one declarative file under version control replaces the README installation section, the Brewfile, the apt script, and whatever Docker setup notes someone half-wrote and never finished. Adoption reflects it: the nixpkgs repository added about half again as many packages in 2024 as it did in 2023, with the steepest growth in machine learning and cloud-native tooling.

The Nix language and the mental model of immutable, hash-addressed packages take real time to sink in, though, and most engineers hit a wall somewhere in the first couple weeks before the payoff starts outweighing the climb. The learning curve is real enough that teams regularly report a significant ramp-up period before the payoff starts to show.

How Nix and Bazel can be composed, and how to choose between them

These two tools solve adjacent problems, not the same one, and conflating them is where a lot of the confusion online comes from. Nix works at the level of the environment and the system: reproducible toolchains, reproducible shells, reproducible OS-level dependencies. Bazel works at the level of the individual build action, doing fine-grained caching and sandboxed incremental compilation. A team can have a rock-solid, pinned environment and still run a build system riddled with sloppy, undeclared per-action inputs.

Composition happens through rules_nixpkgs, which feeds toolchains and environments from Nix straight into Bazel's build graph, so each tool's guarantees reinforce the other's instead of overlapping uselessly. Nix alone makes sense when the goal is environment-level reproducibility: consistent developer shells, hermetic CI, fast onboarding, especially on polyglot teams or in infrastructure-as-code work where the environment is the deliverable. Bazel alone makes sense when the pain is incremental build performance and per-action caching at monorepo scale, and the team is actually willing to invest in writing BUILD files well, which is a bigger ask than it sounds.

Both together tends to show up at larger scale or in security-sensitive pipelines, where a team needs certainty about the CI runner image, the exact compiler version, and the shell utilities in play, and also needs hermeticity down at the level of individual actions. For teams that want most of what Nix offers without swallowing the whole learning curve up front, tools like Devbox put a friendlier interface on top of it, letting a team declare dependencies in a file and know everyone, CI included, runs identical versions. Not a substitute for actually learning Nix. It closes most of the practical gap, though, and for a lot of teams that's plenty.

Hermetic builds as a supply chain security control, not just a reliability optimization

Everything so far has been about reliability. There's a second reason to care, and honestly it's the more urgent one: security. Software supply chain attacks more than doubled globally in 2025, with costs reaching an estimated $60 billion and projected to climb past $130 billion by 2031. Sonatype identified hundreds of thousands of new malicious packages in 2025 alone, pushing the cumulative count past a million. This is not a marginal risk category anymore.

SolarWinds is still the case that makes this concrete at the build-system level. Attackers compromised the build system itself and injected a backdoor into Orion update artifacts; the source repository looked completely clean under inspection, because the tampering never touched the source. A genuinely hermetic, reproducible build would have caught it. Rebuilding from the declared source would have produced a different artifact than the one actually shipped to customers, exposing the tampering on the spot.

The 2024 XZ Utils incident makes a related point from a different angle. A malicious maintainer slipped obfuscated backdoor code in through the build system's test infrastructure, targeting SSH authentication. Same attack surface both times: the build process itself, not the code a reviewer would ever think to scrutinize.

Hermetic builds close these paths structurally, which is the whole point; vigilance is not something you can staff your way into forever. When every dependency has to come from a single trusted control plane with locked-down network access, the build cannot pull from an attacker-controlled registry, silently or otherwise, because there's nowhere else for it to look. When dependencies are vendored or pinned into an immutable snapshot, typosquatting and dependency confusion attacks fail outright, since the build won't resolve to anything outside that snapshot no matter what an attacker publishes under a similar name. And when build instructions live in the repository itself, the build only ever touches trusted configuration and infrastructure, never whatever happened to be sitting on the host that day.

This also makes software bills of materials tractable instead of theoretical. When every input is declared and pinned as a matter of course, generating an accurate SBOM becomes a mechanical output of the build rather than a forensic reconstruction project after the fact, which matters given that most estimates put the open-source share of a typical modern application somewhere between 70 and 90 percent. Brandon Lum, Google's SBOM Lead, put it plainly: the industry needs better build tools that propagate software metadata in the first place. Provenance belongs in the build system, not bolted on afterward as a scanning step.

Regulation is catching up fast too. U.S. Executive Order 14028 and the EU Cyber Resilience Act have both moved SBOMs and build provenance out of best-practice territory and into compliance requirements, and SLSA frameworks now give the industry an actual standard to measure against.

Making hermeticity practical across the development lifecycle

The onboarding thread and the reliability thread turn out to be the same problem wearing two hats. A hermetic environment definition doubles as an onboarding artifact almost by accident: the exact declaration that makes CI reproducible is the same one a new hire activates on their first day. Solving for build reliability and solving for "get someone productive by lunch" end up needing the same file.

One declarative environment spec, checked into version control, replaces the README installation section nobody trusts, the Brewfile nobody updates, the apt script that only works on one distro, and the informal ritual of asking whoever's sitting nearby how they got their machine working. That's the actual shift hermeticity asks a team to make: treat the environment as a versioned, reviewable artifact, the same way the team already treats its source code. It's a smaller ask than it sounds, and a bigger payoff than most teams expect going in.