Environment Reproducibility Auditing in Existing Codebases

"Works on my machine" is not a personality flaw. It is what happens when an organization never declared its environment in the first place, and the audit is how you find out whether that omission is a small gap or a structural one. The rest of this piece maps that audit process, from establishing what your environment claims to be through to fixing what it actually is.

The numbers here are not marginal. Flaky tests tied to environment instability show up in 11 to 27% of tests in large industrial codebases, and account for a significant share of build failures. A 2025 study on automated dependency inference and containerization found that only 25.87% of environments could be successfully reproduced from their declared state, with the rest failing on missing packages, invalid paths, and dependencies nobody had bothered to write down. So the presence of a Dockerfile or a requirements.txt does not mean reproducibility exists; it means the problem has moved somewhere else, usually somewhere less visible.

Reproducibility failures split into two kinds, and knowing which one you have determines the entire remediation strategy. Some environments were reproducible once and drifted, silently, as dependencies updated and nobody re-pinned anything. Others were never reproducible at all; they worked because every developer happened to be running the same ad-hoc setup, and that illusion held right up until someone joined the team with a different machine. Either way, undeclared and unpinned components are not just a reliability risk. They are a supply chain risk, because a dependency nobody tracked is a dependency nobody is scanning for vulnerabilities either.

What a reproducibility audit is actually trying to find

An audit is not a code review. It does not ask whether the logic is correct; it asks whether the environment that runs the logic can be rebuilt exactly, by anyone, on any machine, without tribal knowledge. That distinction matters because teams often conflate the two and end up auditing the wrong thing.

Four kinds of findings tend to surface. Undeclared dependencies: packages or system libraries that live on a developer's laptop but appear in no manifest anywhere. Unpinned or floating versions: a manifest that names a package but not an exact version or digest, which is functionally a promise nobody can verify. Environment drift: the gap between what the documentation says and what is actually running, across developer machines, CI, and production. And missing provenance: no traceable line from a running binary back to the source commit and build inputs that produced it.

Consider that a large-scale academic audit of research replication packages found 64 incomplete or non-runnable artifacts out of 477 examined. Academic teams have a direct incentive to document their environments, since reproducibility is the whole point of publishing a replication package. If that population still fails at this rate, production codebases, where reproducibility is rarely anyone's job description, should be assumed to fare worse, not better.

The audit has a defined scope, and it is worth stating plainly what falls outside it. In scope: language runtime versions, package manifests, lockfiles, base images, system-level dependencies, CI configuration, and environment variables treated as configuration. Out of scope: whether the application logic is correct, whether test coverage is adequate, whether infrastructure provisioning is sound. Those are real concerns, just not this one. The audit produces two things: a gap report describing what's wrong and where, and a remediation backlog ordered by risk.

Assembling a baseline: what the environment claims to be

Before you can find drift, you need to know what was promised. That means gathering every file in the codebase that makes a claim about the environment, and there are more of these than most teams expect.

Start with language manifests: package.json, requirements.txt, Gemfile, go.mod, Cargo.toml. Then check for lockfiles, package-lock.json, poetry.lock, Pipfile.lock, yarn.lock, and note carefully which manifests have a corresponding lockfile and which don't. A manifest without a lockfile is a claim with no enforcement mechanism behind it. Add container definitions (Dockerfiles, docker-compose.yml, devcontainer.json), environment managers (.nvmrc,.tool-versions, devbox.json, shell.nix), and CI configuration: GitHub Actions workflows, GitLab CI YAML, Jenkinsfiles. For each CI file, note what base image or runner it specifies and whether that reference is pinned. Finally, don't ignore the README and any wiki pages; these often contain informal environment instructions that live entirely outside version control and therefore outside any enforcement at all.

Flag floating base image tags immediately. A Dockerfile that pulls node:20 instead of a specific digest offers no reproducibility guarantee whatsoever, since that tag can point to a different underlying image tomorrow. This is a known, well-documented failure mode, not a matter of style preference.

Then check for coverage gaps. Does CI use the same base image as local development, or does it diverge somewhere in the stack? Are system-level dependencies, compilers, native libraries, OS packages, declared anywhere, or simply assumed to be present because they were installed once on someone's machine years ago? Are hardware and driver requirements, GPU drivers in particular, written down anywhere a new team member could find them?

The output of this phase should be a matrix: tool or component, where it's declared, whether it's pinned. This matrix becomes the working document for everything that follows.

Comparing declared state to actual state across environments

Now compare what's declared against what's actually running, across three axes: developer machines, CI runners, and any staging or production environment sharing the codebase.

The comparison itself is mechanical. Run pip freeze, conda env export, or npm list --depth=0 in each environment and diff the output against the lockfile; anything present in the live environment but absent from the manifest is a finding. Check runtime versions explicitly, node --version, python --version, compiler versions, across at least two developer machines and the CI runner, and don't assume they match just because nobody has complained. Inspect CI logs for implicit installs: apt-get install commands, pip installs, or curl | bash patterns that run at build time without being tracked anywhere. Review Dockerfile RUN layers for package installs with no version attached.

Drift tends to hide in predictable places. Developer machines that predate a lockfile's introduction, where the lockfile exists in the repo but was never used to reinstall that developer's environment. CI caches that quietly persist old package versions across lockfile updates, so the CI build looks stable while it's actually running stale dependencies. Operating system differences, macOS on developer machines against Linux in CI or production, surfacing as native library behavior that nobody predicted. And transitive dependencies: the direct dependency is pinned, but what it depends on is not, which means the promise of a pinned manifest is partial at best.

For each discrepancy, record what it is, which environments show it, and whether it's a version mismatch, an undeclared package, or something missing entirely. This granularity matters later when you score risk.

Undeclared or unpinned transitive dependencies deserve a second look here, because they aren't only a drift problem. A component with no declared version has no provenance trail, which means vulnerability scanners can't see it even if they're looking. It's invisible by construction.

Auditing the supply chain layer: from dependency list to provenance

Knowing your dependency versions is necessary. It is not sufficient. A pinned version tells you what you have; it does not tell you whether that's genuinely what you think it is, or where it came from, or whether it's been tampered with somewhere between the registry and your build.

Base images matter more here than most teams assume. Standard public container images typically ship with somewhere around 50 to 60 known CVEs; minimal, source-built images can bring that down to single digits. The base image choice is, in a very literal sense, the highest-leverage decision in a container-based audit, because everything built on top of it inherits its vulnerability surface.

Provenance auditing asks a different set of questions than a plain dependency check. Do the package hashes recorded in the lockfile match what's actually installed, confirming integrity rather than just version number? Are dependencies pulled from a controlled, known registry, or fetched directly from arbitrary URLs with no gatekeeping? Is there a build provenance attestation, SLSA provenance or in-toto, linking the running binary back to a specific source commit and the exact inputs used to build it? And is an SBOM, a software bill of materials, generated as part of the build, or does someone have to reconstruct it after the fact from memory and grep?

That last question matters more than it sounds. Most SBOMs in production today function as compliance snapshots: generated once at the end of a build, filed somewhere, and never opened again. An audit should ask whether the SBOM on hand is actually usable for vulnerability triage or whether it's decorative. Two open standards dominate the format question, CycloneDX and SPDX, and the audit should confirm which one is in use, whether it's machine-readable, and whether it's complete enough to be queried when a new CVE drops.

The March 2025 GitHub Actions incident is worth naming here because it demonstrates the cost of skipping this layer. Attackers modified historical Git tags on a widely used action, one used by an estimated 23,000-plus repositories, and used that access to exfiltrate CI/CD secrets. Any pipeline referencing that action by a floating tag rather than a pinned commit SHA inherited the compromise automatically, with no warning. Pinning to a SHA would have prevented it outright.

That incident should shape the supply-chain-specific items on the audit checklist. Are GitHub Actions or other CI actions pinned to a commit SHA rather than a tag that can move? Is there a secret scanning step in CI that would catch an exfiltration attempt of that kind? And for teams running AI models inside their pipelines, is there documentation of each model's source, version, and training provenance, the ML equivalent of a bill of materials?

Scoring what you find: a risk-ordered gap report

Not every gap deserves the same urgency, and an audit that doesn't rank its findings tends to produce a backlog where cosmetic fixes get done first simply because they're easy, while the dangerous drift sits untouched.

A three-tier structure works well in practice. Critical: undeclared dependencies carrying known CVEs, unpinned base images in anything production-facing, CI actions still referencing floating tags, or manifests with no lockfile at all where one is clearly available. High: version mismatches between CI and developer environments that are actively causing test flakiness, transitive dependencies with no declared version, missing SBOM generation in the build pipeline. Medium: environment instructions that live only in a README, floating version ranges in manifests that do have a lockfile (lower immediate risk, but the drift will recur), system dependencies that are assumed rather than written down anywhere.

For each finding, the gap report should record the component or file affected, which environments show the discrepancy, the risk tier and the reasoning behind it, and a specific recommended action, not "fix it," but the actual step: pin to this digest, add this lockfile, replace this tag with a SHA.

Treat the gap report as a living document, version-controlled alongside the code, updated as remediations land rather than filed away as a one-time deliverable. And use the flaky test rate as your baseline metric: if a significant share of build failures trace to environment instability rather than genuine defects, that's a number you can track downward as remediation proceeds, and a number that gives the effort a measurable return rather than a vague sense of hygiene.

Remediating drift: from lockfiles to fully declarative environments

Start with whatever the gap report marked Critical. Pin base images to digests instead of tags. Add lockfiles wherever a manifest exists without one. Replace floating CI action tags with commit SHAs, following directly from the GitHub Actions incident above.

Lockfile discipline is the foundation everything else builds on. For pip or conda environments, running pip freeze > requirements.txt or conda env export on a clean, known-good machine generates a pinned lockfile from what's actually installed, not from what someone thinks is installed. For npm or yarn, the lockfile needs to be committed to version control, and CI needs to install from that lockfile specifically, not from the manifest alone, or the pinning is theoretical. An uncommitted lockfile should be treated as equivalent to no lockfile at all, because it protects nobody.

System-level dependencies need the same treatment. Every OS-level package the application actually requires should be documented and added to the container definition or environment manager configuration, not left as an assumption baked into someone's laptop. Hardware and driver requirements belong in a machine-readable manifest, not a footnote in a README that a new hire will find on their third day, after already losing two.

Several tools exist for teams moving toward fully declarative environments, and they offer different levels of guarantee. Flox, a single-manifest environment manager built on Nix, is one option that pins dependencies identically across laptops, CI, and production. Dev Containers, the Microsoft specification used by VS Code and GitHub Codespaces, are only as reproducible as the Dockerfile discipline behind them; pin to a digest and the guarantee is strong, leave a floating tag and it isn't. Devbox wraps Nix derivations in a simpler JSON configuration and a familiar CLI, so a command like devbox add nodejs@20 works the same locally and in GitHub Actions, closing the local-to-CI gap without requiring teams to learn Nix directly. Nix and devenv sit at the strong end of the spectrum: identical tool versions on every machine, every time, with no exceptions for "well, it usually works." Devenv, at version 2.2 as of July 2026, covers more than 50 languages and over 100,000 packages across Linux and macOS, x64 and ARM64, including WSL2, plus more than 40 services, and CI runs the exact same environment definition as local development. Nix-based builds can also run sandboxed and containerless, which shrinks the attack surface while keeping full determinism intact.

SBOM generation should move into the build pipeline itself rather than staying a post-hoc exercise; producing the SBOM as a build step, an approach Google's SBOM lead has pushed for, means it reflects the actual inputs of that specific build rather than a best-effort reconstruction weeks later. Enrich each SBOM entry with repository URLs, commit SHAs, and build provenance attestations, SLSA or in-toto, so every component traces cleanly from the running binary back to its source.

Verification is simple to state and easy to skip: re-run the comparison step from the audit on a fresh machine or a clean CI runner. If the environment rebuilds from nothing, with no manual intervention and no one quietly installing a missing package by hand, the remediation is done. If it doesn't, it isn't.

Keeping the audit current: embedding reproducibility checks into the development workflow

An audit is a snapshot. Drift starts again the moment someone adds a dependency without updating the lockfile, and it will happen, because that's the natural failure mode of any manual process left unenforced.

CI is where this gets enforced, not hoped for. Add a step that installs strictly from the lockfile on a clean runner and compares the result against the manifest, failing the build on any divergence rather than logging a warning nobody reads. Make SBOM generation a required build step, not an optional report, and diff SBOMs across builds so new or changed components surface automatically instead of waiting to be noticed. Confirm on every dependency update PR that CI actions are still pinned to commit SHAs, since a single unreviewed change can reintroduce the exact floating-tag risk the March 2025 incident exposed.

On the developer side, the clearest signal of whether an environment is genuinely reproducible is how long onboarding takes. If a new developer still needs three to five days of manual environment configuration, guesswork, Slack messages, someone remembering the one weird flag, the environment is not reproducible in any meaningful sense, whatever the documentation claims. The reasonable target is a single command and roughly two hours or less. Pre-commit hooks and PR checks that catch unpinned additions before they reach the main branch close the loop, turning the audit from a one-time event into a standing property of the codebase.