approaches to managing CUDA and GPU dependencies across AI development environments

CUDA and GPU dependency management is a reproducibility problem, not a packaging inconvenience. Most ML teams treat it like the latter, which is exactly why they lose weeks to it. The NVIDIA stack that supports modern AI work is deeply layered: drivers, the CUDA toolkit itself, cuBLAS, cuDNN, NCCL, TensorRT, and framework-specific optimizations stacked on top of each other — CUDA v13.2 alone ships with over 900 components. A version mismatch anywhere in that chain can take down an entire environment, and ad hoc setup, the kind where someone follows a README and hopes for the best, works until it doesn't. It stops working at the worst possible moment: mid-deployment, mid-experiment, or mid-audit.

Three rules make manual management fragile in practice. CUDA packages have to line up with the installed NVIDIA driver. Every package in an environment has to share the same CUDA toolkit version. Mixing package sources to unlock one specific GPU feature creates conflicts that can eat days of an engineer's time to untangle. Layer onto that the reality that research teams gravitate toward PyTorch for its dynamic computation graphs, production teams standardize on TensorFlow for its deployment tooling, and data science teams keep experimenting with JAX for performance, and an organization ends up running three frameworks with three separate driver compatibility matrices at once.

The pinning requirements get specific fast. TensorFlow Release 18.09 needs CUDA 10 and driver 410.xx. PyTorch, by contrast, needs driver 570 or later on most hardware, though data center GPUs like the T4 get carve-outs permitting driver branches starting at 470.57 (or later R470), 525.85 (or later R525), 535.86 (or later R535), or 545.23 (or later R545). Two frameworks pull their version requirements in opposite directions, and this isn't some niche corner case: over 40,000 companies and 4 million developers build ML and AI work on top of NVIDIA GPUs, so the compatibility question sits underneath a big share of the industry's daily output. Ad hoc setup forces a real tradeoff between chasing the newest package releases for research speed and locking everything down for the stability production needs. Most teams try to do both and end up with neither. That's not a management failure so much as a design flaw in how the problem gets approached from day one.

How environment drift compounds the compatibility problem over time

Version mismatches are the visible failure. Drift is the slower one, and it's worse precisely because it hides.

ML pipelines carry sources of drift that have nothing to do with dependency versions at all. Random initialization, minibatch shuffling order, parallel execution order, and non-deterministic GPU operations all mean the same code can produce different results on different runs, before anyone even touches a package manager. Add hardware variability across workstations, CI runners, and GPU clusters, and reproducibility stops being a fixed target. It becomes a moving one.

Jupyter notebooks make this worse, not better. The format hides state in cells that don't run in the order they're displayed, and it's notoriously loose about declaring what a notebook actually depends on. Research published in Frontiers in Physics found that over 90% of notebooks lack declared dependency versions, meaning the tool most ML practitioners open every single day is also the format least equipped to encode a reproducible environment. Unless a team pins versions explicitly and validates notebooks through CI, that gap just sits there, waiting for the worst possible week to matter.

System-level CUDA installs make the problem worse still, and this one deserves blunt treatment: it's the single most common root cause behind "it worked yesterday" tickets. Installing CUDA at the system level, once the default approach, creates one shared dependency across every project on a machine. Changing it needs root access, and running two projects with conflicting CUDA requirements on the same box becomes close to impossible. The cost shows up downstream: broken CI pipelines, experiments nobody can reproduce six months later, production deployments that quietly diverge from whatever got validated in development. For regulated industries, that divergence isn't a minor headache. Without a traceable record of what actually ran in production, satisfying an audit gets difficult, sometimes impossible.

Conda and lock-file-based tools as a first systematic response

Conda's core trick is resolving non-Python packages, including the system-level CUDA dependencies, as part of the same install process that handles everything else. One command takes you from driver detection through framework installation, no separate manual CUDA toolkit install required.

Three behaviors do most of the work here. Conda reads the installed GPU driver and picks a compatible CUDA version on its own. Installing PyTorch or TensorFlow through conda resolves the complete, tested dependency tree, so CUDA toolkit, cuBLAS, cuDNN, and NCCL all get pulled from one trusted channel, built to match each other. And GPU libraries install through the same workflow as any other Python package, so there's no separate mental model for "GPU stuff" versus everything else.

The lock file that comes out of this process becomes an organizational artifact in its own right. A data scientist can encode exact environment requirements, hand that lock file to a platform team as a deployment spec, and what runs in production becomes traceably the same as what got validated in development. Discrepancies surface instead of hiding.

Anaconda has pushed this further recently. A private preview launched at NVIDIA GTC 2025 added native GPU access through Jupyter Notebooks, building on CUDA Toolkit 12 integration from the year before. As of March 2026, PyTorch, TensorFlow, llama.cpp, ONNX Runtime, and JAXlib are all available with CUDA 12.8+ support, with CUDA 13.1 builds planned as frameworks catch up to the newer toolkit. PyTorch 2.7.0 shipped as Anaconda's first Windows CUDA variant, aimed at GPU acceleration on Windows dev machines, with more Windows packages and Linux ARM64 support planned for 2026. The NVIDIA Nemotron model family now sits in Anaconda's AI Catalyst too, wrapped in the same vulnerability scanning and compliance documentation applied to ordinary Python packages.

Pixi, out of prefix.dev, is worth watching as a newer entrant gaining ground in scientific computing. It produces multi-platform, hash-level lock files covering every dependency from PyPI and conda-forge down to the compiler itself. Researchers can specify hardware acceleration needs across multiple computational environments and solve for a single lock file that works across machines with different GPU types and CUDA compatibility. Presented at SciPy 2025, the pitch is simple: a full software environment spec, CUDA dependencies included, removes the runtime failures that come from missing or mismatched system requirements on remote compute.

None of this solves everything, though, and teams that expect it to are asking the wrong tool to do the next layer's job. Lock-file tools manage packages well, but they stop at the package boundary. They never touch the operating system or the broader system configuration. A conda environment can be perfectly locked and still sit on top of a broken driver install, and no lock file will tell you that.

Fully declarative environments with Nix for deeper reproducibility

NixOS takes a different starting position: not just the application, but the entire system underneath it, AI libraries, GPU drivers, services, configuration files, gets versioned, rebuilt, and rolled back as one reproducible unit.

That buys three concrete properties. Builds are deterministic, so code runs the same way regardless of when or where it's built. Rollbacks are safe, meaning a broken environment gets reverted instantly without manual cleanup work. And deployment collapses to a single command, with the same specification running from a developer's laptop to a full GPU cluster. Nix can even export Docker or OCI images through a build target like nix build .#dockerImage (a name a team defines itself in its flake, not something built in), bridging declarative environment definitions with container-based deployment pipelines teams already run.

The tradeoff is real, and it's worth being blunt about it: Nix's learning curve is steeper than a mainstream Linux distribution, and most small teams don't need to climb it. If three people just want PyTorch and CUDA to install cleanly on three laptops, Nix is overkill, and choosing it there wastes weeks a lock file would have saved. It earns its cost somewhere else: regulated industries, published research that needs to rerun years later, or production deployments spanning multiple architectures, where full-stack reproducibility isn't a nice-to-have but a hard requirement. The properties that make NixOS attractive here, immutability, an explicit dependency graph, consistency across architectures, are also the design principles behind a newer generation of developer-facing tools built on top of Nix.

What supply chain security requires from GPU environment tooling

A modern enterprise application is roughly 80% third-party code, and the CUDA stack is one of the densest concentrations of third-party dependency inside any AI system. Every layer is a door left ajar.

The CVEs aren't hypothetical. PyTorch and TensorFlow have both had vulnerabilities that allowed arbitrary code execution, and the PyTorch supply chain compromise from December 2022 is the case everyone in the field still cites, underscoring how exposed the dependency chain can be. AI work adds risks a traditional software bill of materials was never built to catch: developers pull model weights from registries like Hugging Face and container images from unverified sources, sometimes because an AI coding assistant suggested the fastest path rather than the safest one. Sonatype research found that 80% of dependencies stay un-upgraded for over a year, even after a known vulnerability surfaces.

Standard SBOM formats simply don't capture model weights, datasets, or training configurations, and that gap is specific to how AI systems get built, not some generic supply-chain problem everyone shares. That gap is closing, slowly. SPDX 3.0, released in 2024, added AI/ML profiles with standardized fields for model metadata and dataset references. CycloneDX 1.5 introduced first-class support for machine learning components back in 2023. Together they form the basis for what's now called an ML-BOM, an ML-specific supply chain record, though broad adoption and tooling maturity still lag behind the spec.

Regulators are moving too. The G7 Cybersecurity Working Group published "Software Bill of Materials for AI, Minimum Elements" on May 12, 2026, building on a shared vision the group put out in June 2025, and organized the requirements into seven clusters covering both producers and users of AI systems. The document makes a point of saying an AI SBOM by itself is "not sufficient": it has to connect to vulnerability scanning, security advisories, and tooling that can actually act on what it finds. CISA and G7 partners have worked toward minimum elements for AI SBOMs that go beyond what traditional software bill of materials formats were built to capture. On the EU side, Article 11 of the AI Act requires technical documentation for high-risk systems, covering design, development, and risk management per Annex IV, and NIST published SP 800-218A as a companion profile to its Secure Software Development Framework, explicitly extending it to AI and ML components rather than folding it into the original SSDF v1.1.

The broader software supply chain security conversation puts all of this under one roof. The CUDA environment sits inside that perimeter whether a team treats it that way or not, and an environment spec that only serves as a runtime convenience isn't enough anymore. It has to double as an auditable provenance record, and that's exactly what lock files and Nix derivations already produce, almost as a byproduct of doing their normal job.

Why GPU environment complexity is an onboarding and platform engineering problem

New hires without structured onboarding take significantly longer to become productive, and for AI teams where the environment itself is GPU-dependent, unstructured onboarding drags that timeline out even further. A new engineer who spends their first week fighting driver mismatches isn't learning the codebase, and nobody should mistake that week for ramp-up time.

Platform engineering has grown fast enough to actually fix this at scale. According to a Gartner forecast, 80% of large software engineering organizations are expected to run dedicated platform teams, up from 45% in 2022. That's the infrastructure needed to solve GPU onboarding systematically, and it already sits inside most large organizations. When self-service environment provisioning is done well, onboarding timelines that once stretched across weeks can collapse to hours.

Self-service GPU onboarding, done properly, needs a few specific things: a single command that handles driver detection, CUDA version resolution, and framework installation, replacing the multi-step README full of footnotes and caveats. Environment specs checked into version control next to the application code, so a new contributor inherits the right GPU environment automatically instead of assembling it by hand. And the same spec working across a developer's laptop, a CI runner, and a GPU cluster without manual per-environment tweaks.

The platform team's job here isn't to mandate one tool for everyone, and teams that try that usually end up fighting their own developers instead of the actual problem. The real job is to publish base GPU environments that individual teams can extend, which is the difference between bottlenecking developers with rigid, one-size-fits-all tooling and giving them defaults flexible enough to build on. Increasingly, the "developer" needing that consistency isn't only human, either. AI agents running CUDA-dependent workloads carry the same version alignment requirements people do, and ad hoc setup breaks agent pipelines exactly the way it breaks human ones.

Choosing an approach based on what your team actually needs to guarantee

None of these approaches is wrong on its own terms, but they guarantee different things, and picking one means being honest about which guarantee the team actually needs, not which tool has the most buzz this year. Most teams pick based on buzz anyway, which is how a five-person research group ends up debugging Nix flakes when a conda lock file would have covered everything they needed. That's the wrong call, and it's worth saying plainly: reach for Nix only once a specific hard requirement (audit trail, multi-architecture deployment, research that has to rerun years later) actually exists. Absent that, it's the wrong tool.

Conda and Pixi solve the package-level alignment problem with the least friction to adopt. Lock files buy auditability and portability without asking a team to overhaul its infrastructure, which makes this the right fit for teams that need a reproducible Python-plus-CUDA stack across a mix of hardware, nothing more elaborate than that. Container-based environments, GPU-Jupyter or NVIDIA's CUDA Docker images among them, isolate the whole runtime and work well for shared infrastructure through something like JupyterHub on Kubernetes. They guarantee consistency across users, but they push the reproducibility question up a level, onto image management, rather than actually resolving it. Nix-based declarative environments guarantee reproducibility across the entire stack, system configuration included, and that's the guarantee that matters when regulatory audit trails, multi-architecture deployment, or long-term research reproducibility are non-negotiable. That guarantee costs more to adopt, and teams that reach for Nix without one of those hard requirements are usually paying for reproducibility they don't need yet.

Lock files encode reproducible environments as organizational artifacts in their own right: a data scientist pins CUDA and framework versions exactly, hands the manifest to the platform team, and production ends up running the same stack that got validated in development. Tools like Flox build on that same model, using a single manifest to manage CUDA dependencies alongside the rest of an environment, so the GPU stack stays consistent across laptops, CI, and production without a manual toolkit install or driver reconfiguration at each stop along the way.

Whichever path a team takes, the SBOM and provenance requirement doesn't go away. Lock files, Nix derivations, and container manifests can all satisfy it, but only if someone actually configures the toolchain to produce and store the record. It never happens on its own, no matter what a vendor's pitch deck implies.

A few questions are worth answering honestly before choosing. Does the same environment need to run across multiple architectures today (x86, ARM64, data center GPUs), or is that a future problem someone else can solve later? Is the team in a regulated industry where the environment itself doubles as an audit artifact? How many frameworks run side by side, and do their CUDA requirements actually conflict, or just look like they might on paper? And what does a broken CI pipeline or a failed experiment reproduction actually cost, set against the cost of adopting stricter tooling today?

The principle underneath all of it doesn't change based on which tool wins. Environment composition has to be explicit, version-controlled, and activated automatically. The moment GPU environment setup depends on a person making a judgment call on some individual machine, drift has already started, and it doesn't reverse itself on its own.