Reproducible Environments for AI Coding Agents

Agent-generated code fails more often than most teams assume, and the reason is structural, not incidental. A 2025 empirical study (arXiv:2512.22387) tested Claude Code, OpenAI Codex, and Gemini across 300 generated projects, built from 100 standardized prompts in Python, JavaScript, and Java, and found that only 68.3% ran successfully in a clean environment straight out of the box. Anyone treating agent output as production-ready by default is working from a number that does not hold up, and teams that keep doing it anyway tend to find out why that number matters in a CI runner at 2am.

The language breakdown makes the point sharper. Python projects succeeded 89.2% of the time. Java projects succeeded only 44.0% of the time, worse than a coin flip, and the gap tracks closely with ecosystem complexity: Java's richer build tooling gives an agent more places to state something true on its face and wrong in execution. The same study found that even the best-performing agent could reproduce published research paper code only 54.1% of the time, a useful reminder that "released" and "runnable" are separate claims. Scale a 31.7% baseline failure rate across dozens of agent-generated projects a day, and what looked like an occasional annoyance for one developer becomes a systemic tax on the whole engineering org.

The hidden dependency gap that makes agent-generated code structurally unreliable

The same study measured something more specific than a pass/fail rate: a 13.5x average gap between the dependencies an agent declares and the dependencies the project actually needs at runtime. Import scikit-learn and the agent writes one line into requirements.txt. What actually loads at runtime is closer to 52 packages, once everything scikit-learn pulls in underneath it gets counted. The agent wrote the line and left the graph beneath it alone.

This looks more like a structural limitation than a generation flaw, and the distinction matters. The gap between what an agent declares and what actually runs suggests models learn to write import statements without accounting for the full closure of everything those statements drag in at runtime.

The study's authors put it plainly: until agents learn to specify complete execution environments and not just code logic, they function as sophisticated autocomplete rather than autonomous contributors. That distinction should settle how much trust agent output gets, and the honest answer is less than most teams currently give it. Agent-generated code cannot be treated as deployable the moment it is written; every handoff from agent to CI carries a reproducibility gap that needs checking, not assuming away. Human developers have always had a version of this problem, but nobody sits at a keyboard anymore noticing, mid-task, that the environment feels off before committing.

How the same drift that breaks human workflows breaks agent workflows faster

Environment drift is an old problem: different tool versions across machines, OS-level quirks, setup steps nobody wrote down. For agents, the equivalent is undeclared dependencies, package versions that float instead of pin, and no clean-room test run before the code gets handed off. The mechanism is familiar, but the speed has changed, and speed is the whole story here.

A human developer who hits a missing dependency tends to notice it directly and can investigate before moving on. An agent runs continuously, and its failures happen at generation speed, not human speed. Volume comes first: an agent generates and commits code faster than any review process can catch bad environmental assumptions. Silence follows, because where a human stops and asks a question, an agent tends to retry, or produce a fix that looks plausible, or generate code that runs fine inside its own context window but fails the moment it hits a clean shell somewhere else. Propagation finishes it: agent output often lands straight in CI, so a wrong environment assumption surfaces downstream in a pipeline, far from where it is easy to trace back to the original bad guess.

Analysis of agentic development environments from October 2025 makes a related point directly: sandboxing the agent covers only half the job. The other half is making sure the agent has an environment that actually reproduces correctly when it runs in cloud infrastructure, not just in whatever local context it was built for. Guidance on running production AI agents echoes this from a different angle, arguing that reproducibility has to cover every layer an agent touches — model weights, prompt versions, tool definitions, and runtime configuration — alongside the lines of code the agent outputs. The problem is structural and gets worse with scale, so the fix has to be systematic too, the same discipline production engineering teams already apply to their own services. They just have not extended it to agents yet.

What systematic environment rigor actually looks like in practice

The core idea is not complicated: treat the agent's environment the way a production service gets treated. Declared. Versioned. Locked. Testable on its own, in isolation, before it touches anything else.

The tooling to do this already exists, and picking one matters less than actually using one consistently. Flox, for instance, is a single-manifest environment manager built on Nix that runs the same declared environment across laptops, CI, and production. Nix and Nix flakes pin every dependency, including the package manager itself, and are designed to produce consistent environments across different machine architectures. The tradeoff is a steep learning curve, and that is the practical reason Nix has not seen wider default adoption despite its strong reproducibility properties. Devbox, built by Jetify, wraps Nix behind a plain devbox.json file and a CLI that reads like a normal package manager, so a command like devbox add nodejs@20 replaces writing a Nix derivation by hand. Its lockfile pins exact versions and transitive dependencies, it generates a devcontainer.json for VS Code, and it is built explicitly to run the identical environment locally and in CI through GitHub Actions. Dev Containers, the spec Microsoft originated and that VS Code, JetBrains, and GitHub Codespaces have all adopted, define an environment through a Dockerfile plus a devcontainer.json, and they suit agent sandboxing particularly well because the container boundary doubles as a security boundary. mise handles runtime version pinning with a lighter setup overhead, suited to teams that need version management without a full environment specification.

Here is where most teams get it wrong: the tool choice matters less than what happens after it's picked. Teams often use a floating tag, something like node:20, which can point to a different image entirely the next time it gets pulled. Most teams tag their images without pinning to a digest, so the environment drifts quietly over months and nobody notices until something breaks. Locking the environment in code, and running that exact lock file in CI as well as locally, is the principle that actually matters, more than any tool choice. Do that, and the entire "passes on my machine, fails in the pipeline" category of bug, the one agent workflows inherit by default, mostly disappears. For agent-specific setups, devcontainers fit especially well, because the sandbox boundary and the reproducibility boundary are the same boundary, and the spec has broad adoption across major development platforms.

Why CI/CD is where agent environment failures do their real damage

CI is where a small, local assumption turns into a repeated, expensive failure. An agent that generates code with a broken environment assumption does not just fail once. That assumption gets pushed into the pipeline and fails again on every runner, every pull request, every deployment attempt, until someone finds and fixes the root cause.

One in five organizations reported a security incident tied to their CI/CD pipeline in the past year, and agent-generated code entering those pipelines adds further surface area on top of that existing risk. The failure pattern is mechanical and repeats the same way each time. The agent writes code carrying an implicit dependency assumption. A human reviewer checks the logic, not the completeness of the environment, and approves it. The CI runner turns out to have a slightly different base image or tool version than what the agent assumed. The build either fails outright or, worse, succeeds while quietly resolving to a different dependency version than the one the agent had in mind.

The fix means making the environment itself a first-class artifact, something the agent declares and CI checks against, rather than something CI has to infer. When the same devbox.json or devcontainer spec governs both the local environment and the CI runner, sharing one lock file across both, an entire class of pipeline failure has nowhere left to happen. No serious engineering team ships to production without a declared, tested environment spec behind it. CI deserves that same discipline, and so does every agent feeding code into it.

How agent workflows expand the software supply chain attack surface

The baseline threat here predates agents entirely. In March 2025, StepSecurity disclosed that attackers had modified historical Git tags on a widely used GitHub Action, one relied on by an estimated 23,000-plus repositories, exfiltrating CI/CD secrets into public Actions logs. Agents stack three more surfaces on top of that existing risk.

Generated dependencies carry no provenance, for one. An agent declaring a package name says nothing about that package's integrity, its true origin, or whether it even resolves to the intended package under every possible package manager configuration. The AI model itself is now a supply chain component too, not just the code it produces; traditional scanners cannot read inside a model file, and pickle-format models carry a known risk of remote code execution simply from being loaded. The same logic that pushed teams toward maintaining SBOMs for code now applies to models, and teams need ML-BOMs that document training data, architecture choices, and safety benchmarking. Then there is the MCP layer, the piece that lets an agent talk to outside data and tools, and it is its own governance problem entirely. An unverified MCP server is functionally an unverified third-party dependency, except one that often carries privileged access into systems the agent touches.

None of this happens in a vacuum of underused defenses, either. Standard base container images typically ship carrying somewhere between 50 and 60 known CVEs, while minimal, source-built images bring that down to single digits, a lever most teams have available and mostly do not pull. Regulation is tightening regardless of how agent adoption plays out: U.S. Executive Order 14028 requires SBOMs from federal software vendors, and NIST SP 800-218 lays out secure development practices across the full software development lifecycle. Teams building with agents do not get an exemption from either one. Across all three new surfaces, the fix stays the same: provenance has to get built into the environment declaration itself, not reconstructed after the fact from whatever artifacts happen to survive the build.

Why SBOMs generated after the fact don't solve the agent provenance problem

Most SBOMs get generated at the tail end of a build, filed away, and never looked at again. That is the whole problem in one sentence. A 2026 analysis of the practice put it bluntly: there is a hard ceiling on how much value a list like that provides if nobody acts on it. Producing the document is one step. Using it is a separate, often skipped, one.

That frustration showed up directly at the July 2025 S3C2 Government Summit, where twelve participants representing six US agencies said much the same thing: the SBOM existed, but without a compliance mandate forcing someone to act on it, nobody knew how to turn the document into actual change. A systematic review applying the ISO/IEC 25019:2023 quality model found that most SBOM shortcomings cluster into three buckets: the data is not trustworthy, the format is hard to generate and share, and the content is not suited to real risk assessment or vulnerability triage in the first place.

For agent workflows, a post-build SBOM misses the moment where the real problem starts. The agent declares its dependencies during code generation, not afterward, and if that declaration is incomplete, which the 13.5x expansion gap says it usually is, the SBOM built downstream inherits that same incompleteness. It documents a gap instead of closing one, and that ordering problem is the entire argument against treating SBOM generation as a compliance step bolted on at the end. Google's SBOM lead, Brandon Lum, has argued for building software metadata as a natural byproduct of the build process itself, so a project's full composition is discoverable right at the source instead of guessed at downstream. SBOM entries also need enriching with repository URLs, commit SHAs, and build provenance attestations, using open standards like in-toto, SLSA provenance, CycloneDX, or SPDX, so a binary can get traced back to its source without locking a team into one vendor's tooling. For agent environments specifically, the environment spec itself, whether that is a devbox.json, a flake.lock, or a devcontainer definition, is the natural anchor point for SBOM generation, arising directly from building the environment correctly instead of getting tacked on afterward as a checkbox.

What a rigorous agent environment workflow looks like end to end

Treat the agent like a new hire who needs onboarding into a declared, versioned environment. That framing changes what the workflow has to include, and it is worth taking literally rather than as a metaphor.

Start with environment declaration: the project's runtime, system dependencies, and tool versions live in version control next to the code, and the agent operates inside that declared spec instead of improvising around it. Add lockfile discipline on top, so transitive dependencies stay pinned and any change to the dependency graph shows up as a visible, reviewable diff instead of a silent expansion. Before any agent-generated code reaches CI, run it through clean-room verification, a fresh instantiation of the declared environment, which catches the roughly 31.7% of projects that fail out of the box before that failure ever touches the pipeline. Keep CI parity by running the exact same environment spec in CI runners as in local and agent contexts: one lock file, three places it runs, no drift between them. Make provenance build-native, so SBOM generation happens as a step inside the build itself rather than a scan run afterward, with attestations attaching to every artifact the agent produces. And govern MCP servers and model versions the same way, as declared dependencies with their own provenance requirements, not trusted defaults nobody checks.

The tools already line up with this pattern. Nix-based setups, Devbox included, naturally produce the lockfile this depends on and can anchor SBOM generation directly. Dev containers supply the sandbox boundary. SLSA and in-toto handle the provenance attestation layer. Whoever owns the base environment spec ends up making agent workflows faster, because the agent inherits a known-good starting point, and any deviation from it becomes explicit instead of silent. Research on employee onboarding (SHRM, 2025) found that structured onboarding meaningfully improves both retention and productivity, and the same logic holds here. An agent dropped into a well-declared environment gets productive right away, while one dropped into an undeclared environment produces something close to that 31.7% failure rate as a matter of course.

Where teams should start if their agent environments are currently undeclared

Most teams adopting AI coding agents in 2026 built their environment infrastructure years before agents existed, so none of it was designed with this problem in mind. The real question is where fixing it first buys the most. The base environment goes first, because everything downstream depends on it holding still.

Lock the base environment before anything else. If a team does nothing else, pinning the runtime, the system dependencies, and the tool versions that every agent-generated project inherits closes off the single largest source of drift, ahead of lockfiles, CI parity, or provenance tooling entirely. Everything else in this piece, the clean-room checks, the shared CI lock files, the build-native SBOMs, depends on that base layer holding steady first. Get that piece declared and versioned, and the rest of the workflow has something solid to stand on. Skip it, and every later fix patches a foundation that is still moving underneath it.