Declarative Environment Definitions and Hermetic Builds
A hermetic build runs in complete isolation from the host machine, takes only explicitly declared inputs, and produces identical outputs no matter when or where it runs. Most teams that claim to have hermetic builds don't, because they skipped the one step that actually makes hermeticity possible: writing down, completely and in advance, every dependency, tool, and assumption the build touches. Isolation without a full declaration is a sandbox with a hole in it, and that hole is where the surprises come from six months later.
Three properties have to hold at once for a build to earn the word hermetic. Determinism means the same inputs produce the same outputs every time, with no timestamps, no random seeds, no system-dependent file paths sneaking into the artifact. Isolation means the build process can't reach outside its declared boundary, no implicit system libraries, no ambient environment variables leaking in from whatever machine happens to be running it. Explicit inputs means every dependency, toolchain, config file, and OS-level assumption gets named and versioned before the build ever starts. Drop any one of the three and the other two stop meaning much.
Isolation is the property people reach for first, and it's also the one that fools them. A sandboxed process that still pulls in an undeclared system Python, or resolves a dependency tagged "latest" instead of a pinned version, is isolated in name only. The sandbox walls are real; the declaration is missing, and no amount of enforcement fixes that gap after the fact. A shell script that installs things in order is a recipe. A manifest that states the final, complete set of inputs is a contract. Nobody flips a switch on a build tool and gets hermeticity handed to them. Teams earn it by being exhaustive about declarations before any sandbox gets involved, and most find that out the hard way, usually mid-incident.
How ad-hoc environment setup breaks the hermetic contract, step by step
Picture the typical setup: a README with a numbered list of install steps, an implicit assumption that the developer's machine already has half the toolchain sitting on it somewhere, and a tangle of globally installed packages and project-local ones that nobody has fully mapped. It works fine, until it doesn't.
The break happens in predictable places. Undeclared system dependencies are the classic one: a build succeeds on one machine because OpenSSL 1.1 happens to already be there, and fails on the next machine where it isn't, with no line in any manifest that would have flagged the difference. Version ranges and "latest" tags cause a subtler version of the same failure; a dependency upgraded overnight changes a build's behavior without a single declared line of configuration changing, so the failure shows up disconnected from any commit anyone actually made. Ambient environment variables, PATH, PYTHONPATH, JAVA_HOME, set differently across machines mean what everyone calls "the same build" is quietly running different toolchains depending on whose laptop it landed on that week. Manual patches accumulate silently in hosted environments too: someone SSHes in, patches a library to unblock a deploy at 11pm, and there's no record anywhere of what changed or why.
This is configuration drift, and it's the mechanism, not just a symptom of something else. Two environments assumed to be identical diverge one undeclared difference at a time, and each difference becomes a hidden variable sitting inside the build. A study of a large sample of Docker builds, published January 2026, found that only a small fraction of rebuilt images matched the original installed package versions exactly. Docker is a format built specifically to contain this kind of drift, and it still fails at this rate when the underlying declarations are incomplete. Containers reduce some sources of drift, sure, but the discipline of declaring inputs has to come before the tooling choice, never after.
Why a declarative manifest is the prerequisite, not the finishing touch
You cannot isolate what you haven't named, and that's the whole problem in one sentence. A build tool can only sandbox the inputs it knows about. Any input it doesn't know about either fails the build outright or, worse, leaks in quietly from the host and starts behaving as if it belongs there.
A complete manifest covers application dependencies pinned to exact, content-addressed versions rather than ranges. It covers compiler and interpreter toolchains specified explicitly rather than inherited from whatever happens to be on the system. It covers build scripts and configuration files treated as versioned inputs rather than assumptions baked into someone's muscle memory, and OS-level details, locale, timezone, kernel headers, made explicit instead of left to whatever the host defaults to that day. Flox, for instance, manages this through a single manifest.toml that pins the same dependencies across laptops, CI, and production.
Nix is a useful worked example here. Nix flakes make builds hermetic by pinning every dependency to a concrete revision of nixpkgs; the flake file is the manifest, and the sandbox that runs the build enforces that nothing outside it is reachable. A hermetic build behaves the way a pure function does in functional programming: outputs depend only on declared inputs and nothing else. Any implicit dependency is a side effect, and side effects break purity, full stop. The Reproducible Builds project's own diagnosis of common failure points, paths, locales, timezones, environmental assumptions, lands on exactly this class of undeclared input. These things almost never show up in a README's install steps, which is exactly why nobody controls them, and exactly why builds diverge in ways no one can trace later.
Writing the manifest first is the design work itself. It isn't overhead layered on top of the real engineering.
How Bazel and Nix enforce the declarative-to-hermetic pipeline in practice
Bazel and Nix arrive at the same property from two different directions, and which direction a team takes shapes how the discipline actually sticks.
Bazel starts from the build graph. Every action declares its inputs and outputs explicitly in BUILD files, and the sandbox, a Linux namespace or chroot, enforces that only those declared inputs are visible to the process. Bazel's cache keys are cryptographic hashes of the inputs, the environment, and the rule definitions themselves, so a cache hit is a guarantee rather than a hopeful guess. Nix starts from the environment definition instead: a package derivation declares every dependency down to libc, the Nix store enforces isolation through content-addressed paths, and nothing already built gets mutated afterward.
A 2024 Bazel user survey from Aspect.dev found teams using Bazel's Python rules cut build variability by 95% and reduced debugging time tied to reproducibility issues by more than 70% in large codebases. Read those numbers carefully; they're measuring the gap between what a build declared and what it actually touched. Closing that gap is the real work. Bazel enforces the boundary, but it can't do the declaring for you, and neither can Nix. devenv illustrates the Nix approach at the scale of a developer's daily environment rather than a single package build: its lock file pins every dependency to a concrete Nix store path, and that same lock file produces native binaries on ARM macOS and on x86-64 NixOS with zero adaptation required. CI runs the identical environment by executing one command, not by re-deriving it from some parallel set of instructions someone wrote separately.
Neither tool can infer an undeclared dependency out of thin air. Both simply refuse to let one in through the back door. Teams that adopt Bazel or Nix without doing the exhaustive declaration work first end up with a sandbox but not hermeticity, and that gap between form and substance is common enough to name plainly. This is the reason so many "hermetic" build migrations quietly stall six months in, once the novelty wears off and the undeclared dependencies start surfacing one by one.
What hermetic builds produce beyond a binary: SBOMs and provenance as automatic outputs
Here's the part that doesn't get enough attention. Because a hermetic build already requires every input declared and versioned before it runs, it already has everything needed to generate an accurate software bill of materials. The SBOM is a byproduct of a manifest that was always going to exist anyway, generated alongside the build instead of bolted onto the end of it as an afterthought.
The compliance pressure behind this is real and isn't going away. U.S. Executive Order 14028, signed May 2021, requires federal software vendors to produce SBOMs, and NIST SP 800-218 lays out secure development practices across the software lifecycle that assume this kind of artifact exists. Most SBOMs generated today are compliance snapshots, though: produced once at the end of a build, filed somewhere, and never opened again. Analysis tied to ISO/IEC 25019:2023 finds the bulk of SBOM shortcomings fall into three buckets: they aren't trustworthy, they aren't usable, and they aren't suited to the security tasks people actually need them for, like vulnerability triage.
Google's SBOM lead, Brandon Lum, has argued the industry needs build tools to propagate software metadata directly, rather than asking downstream consumers to reverse-engineer composition after the artifact already exists. The Yocto Project's OpenEmbedded build system, which now generates SBOMs natively as part of the build itself, is one working example of that approach. Provenance standards close the remaining loop; in-toto and SLSA attestations tie a binary back to the exact source commit and build process that produced it, and an SBOM enriched with repository URLs, commit hashes, and build attestations gives you a real chain from artifact back to source, not just a list of package names with no history behind them.
None of this is abstract risk management. Supply chain attacks rose 742% annually between 2019 and 2022, and in 2025, attackers modified historical Git tags on a GitHub Action used by an estimated 23,000-plus repositories and used it to exfiltrate CI/CD secrets. That attack surface, an external input nobody re-verified, is exactly what hermetic builds close off. Teams doing this well lean on standard formats, CycloneDX, SPDX, in-toto, SLSA, OSV, VEX, rather than proprietary schemes, so the evidence can actually be shared with customers and regulators without locking anyone into one vendor's tooling.
The container image as the place where undeclared inputs do the most damage
Container base images are where the undeclared-input problem does the most visible damage, and it's worth sitting with why. A standard public base image ships with a long list of packages the application never asked for, never declared, and in most cases never even inventoried. Standard public images typically carry somewhere in the range of 50 to 60 known CVEs, while minimal, source-built images, where every included package is a deliberate, declared choice, cut that down to single digits.
This is a declaration problem before it's a patching problem, and the order matters. A CVE in a package nobody declared as a dependency can't be governed, can't show up correctly in an SBOM, and can't be removed by a team that doesn't know it's sitting in the image in the first place. The undeclared package is the vulnerability, independent of whether a patch even exists for it.
"Source-built" means starting from a minimal, declaratively defined base where every component is an explicit decision, rather than inheriting whatever the upstream image maintainer decided to bundle in that week. This is a platform engineering problem as much as a security one. Teams that define base images declaratively, treating the image definition as a versioned artifact that goes through review like any other code, hold every service to one known-good baseline instead of letting each team accumulate its own private pile of undeclared packages. Development-time environment definitions and build-time container definitions have to converge here. Any inconsistency between the two isn't a minor mismatch; it's a fresh source of drift, sitting in exactly the spot the last one got patched out of.
Why environment drift between dev, CI, and production is the practical enemy of hermeticity
Dev, CI, and production get treated as three copies of the same environment. In practice, each accumulates its own undeclared differences: a different OS point release here, a different system library there, a tool one engineer installed on a Tuesday for a reason nobody wrote down and everybody's since forgotten.
The consequences aren't hypothetical. A 2023 analysis spanning 780 runs across four software containers and thirteen hardware types found test-accuracy drift of several percentage points for binary classifiers and even higher for LSTM models, caused entirely by environment differences, with zero change to the model or the code responsible. A test suite that passes in CI and fails in production, or the reverse, is a signal about the environment, not the test. It means the two are running different implicit inputs, and someone just hasn't found which one yet.
The Reproducible Builds project's guidance for closing this gap reads almost like a checklist: pin important dependencies to immutable versions or digests, rebuild environments on a defined schedule rather than letting them sit indefinitely, and test clean rebuilds regularly rather than trust that yesterday's build still reflects today's environment. In practice this discipline lives in artifacts like devenv.lock files or Nix flakes, mechanisms that span dev, CI, and production with a single locked definition. When all three consume that same locked environment, drift stops being something a policy discourages and becomes something the infrastructure makes structurally impossible.
"Works on my machine" often gets treated as a communication failure, something a Slack message or a better README would fix. The deeper cause sits in infrastructure: the absence of one shared, authoritative declaration every environment is required to honor, with no exceptions carved out for convenience.
How declarative environments compress developer onboarding from days to a single command
A multi-step README with a handful of caveats scattered through it isn't an onboarding process. It's a sequence of opportunities for the new hire's environment to diverge from everyone else's, before they've written a single line of code.
The cost of getting this wrong is measurable, and it's larger than most engineering leaders assume. SHRM reported in 2025 that new hires without structured onboarding reach only about 25% productivity in their first thirty days, while Stack Overflow's 2024 Developer Survey found companies with structured onboarding see new engineers reach productivity 62% faster than those without one. One practitioner case study makes the curve concrete: average time-to-first-commit dropped from 12.4 days in 2023 to two hours and eleven minutes by December 2025, the result of an internal developer platform that treated environment provisioning as a problem to solve once, rather than a ritual every new hire repeats from scratch.
Once the environment definition lives as a versioned file in the repository, onboarding stops being a process someone has to document and becomes a command someone runs. The definition enforces correctness in a way no README, however carefully written, ever could. Retention numbers back this up: SHRM's 2025 data puts early attrition at 22% of developers leaving within their first ninety days when onboarding is unstructured, against 82% higher retention where the process is solid. Friction in week one is a leading indicator of who stays, not some minor inconvenience to smooth over later once things settle down.
This is also why platform engineering has grown from a niche discipline into something most large engineering organizations now staff deliberately. Gartner reported in 2026 that a large majority of large software engineering organizations have a dedicated platform team, up from fewer than half in 2022. Done well, a platform team's job is to produce declarative base environments that developers activate rather than configure from scratch, giving teams a good default to build on instead of a gate they have to clear before they're allowed to be useful.
AI coding agents expose the same undeclared-dependency problem at a different layer
The first time I watched an agent fail this way, it took me a minute to place what I was even looking at. AI coding agents, Cursor, GitHub Copilot, Claude Code, and tools like them, are environment consumers in exactly the same sense a human developer is. They execute code, run test suites, call build systems, and depend on the correctness of that environment just as much as a person does. When the environment is wrong, they fail the same way a person would, just faster, and with a lot less context about why.
What's actually new is the shape of the undeclared dependency. An AI model is itself a third-party dependency, but it's one a conventional dependency scanner has no vocabulary for at all. It has a version, a training provenance, known behavioral quirks, and none of that shows up in a standard manifest built for tracking library versions and not much else. Pickle injection makes this concrete rather than abstract: many models ship in formats, pickle being the common example, that allow arbitrary code execution the moment the model loads. An undeclared model dependency at that point becomes a security vector, functionally identical to pulling in an unverified package with no version pin and no audit trail behind it.
The natural extension is an ML-BOM: a declarative manifest that lists model dependencies with the same rigor a software bill of materials applies to code, versions and provenance included, so the model an agent loads is as accountable as the compiler it calls. The layer changed. The problem didn't: an input nobody wrote down, governing behavior nobody can fully predict.
