Container Image Reproducibility and Provenance Attestation

The evidence against naive reproducibility is empirical. A 2026 systematic study rebuilt 5,298 historical container images from 2023 GitHub Actions workflows and found that Docker guarantees reproducibility under no tested definition; no set of Dockerfile rules produces it either. Separately, 9.81% of Dockerfiles drawn from 8,132 candidate projects exhibited flaky build behavior, and the authors noted that real-world rates would be higher because the sample skewed toward high-quality projects. These aren't anecdotal failures. They are systemic, and understanding the distribution of failure modes determines how you respond to them.

Non-determinism enters through two distinct channels, and conflating them produces interventions that only partially work. Infrastructure-level sources include build tooling that embeds timestamps in image layers, cache invalidation that forces re-resolution of packages against live registries, and layer ordering that varies across build workers. These are largely invisible to the developer writing the Dockerfile and require tooling-level intervention to suppress. They also get blamed on developers anyway, which is part of why they persist. Developer-controlled sources are a separate category: floating version references that resolve to different packages across time, ad-hoc cache or log entries written into an image during a hotfix, build arguments that vary by machine. Removing infrastructure sources of non-determinism leaves this second category entirely intact.

The metadata problem is underappreciated in most conversations about this. Roughly 15% of reproducibility failures trace to embedded build dates alone, per the same 2026 study. A timestamp is a reproducibility defect wearing metadata's clothes.

Drift compounds organizationally in ways that are genuinely difficult to detect after the fact. A hotfix applied directly to a production image, an environment variable injected during an incident, a base image silently rebuilt upstream with different package contents while keeping the same tag — each change is small, undocumented, individually invisible. Together they constitute a provenance gap. Organizations deploying directly from container registries rather than rebuilding from source are particularly exposed, because any compromise or silent change in the build or publication pipeline propagates downstream without a rebuild triggering detection.

Pinning a tag or digest controls the named artifact. It doesn't control what produced it. Upstream base images can be rebuilt with different contents and published under the same tag. Independently rebuilding official base images from their published source doesn't produce a bitwise match with the upstream version. Reproducibility is a property of the entire build process, including its inputs, environment, and toolchain. It is not a property of Dockerfiles or tags, and no checklist closes these failure modes. The conversation has to move to verifiable provenance.

What Provenance Attestation Actually Means in a Container Context

Provenance is verifiable information about where an artifact was produced, when, and how — the identity of the build system, the source repository commit, the build steps executed, the toolchain versions used. The word "verifiable" carries all the weight in that sentence. Asserted provenance, where a developer or build system claims certain inputs were used but the claim can't be independently confirmed, provides weak guarantees. An attestation is different in kind: a signed, machine-readable document attached to the image in the registry, structured so that any downstream consumer can verify the claims without trusting the publisher.

Reproducibility and provenance are not interchangeable, and neither substitutes for the other. You can rebuild the same bits without being able to prove to a downstream consumer that those bits came from a trusted process. You can have a signed record of a build without being able to independently verify that record by reproducing the artifact, which renders the attestation unenforceable against the actual image. The tooling to address both problems exists. The organizational discipline to deploy it consistently is the actual gap.

Mechanically, provenance attestation in the container ecosystem works as follows. The provenance document is signed, typically via Sigstore's keyless signing mechanism, which combines a Fulcio certificate for build identity with a Rekor transparency log entry for tamper-evident timestamping. The signed attestation is pushed to the registry alongside the image digest as a cosign attestation. The artifact anchor for this entire chain is the content-addressable SHA-256 digest of the image. Tags cannot serve this role; they are mutable references that can be reassigned to different images. A digest is cryptographically bound to the specific bytes of a specific image, and an attestation referencing that digest is bound to exactly that artifact and no other. Every downstream verification step, every policy decision, every audit trail depends on the integrity of that anchor.

How SLSA Levels Translate to Concrete Build Pipeline Requirements

The Supply-chain Levels for Software Artifacts framework, maintained by the Open Source Security Foundation, provides the operational vocabulary for provenance rigor. SLSA is a graduated set of security levels designed for progressive adoption. SLSA v1.2, released in November 2025, formalized the Build Track as the first fully defined track and represents the current target for most container-focused implementations.

The levels are not labels. They are material requirements with consequential differences between them.

Level 1 requires that provenance exists and documents the build process. Even unsigned, this raises the floor. A team with Level 1 provenance has articulated what produced an image; a team without it hasn't. The gap between assertion and verification remains open, but the artifact of record exists.

Level 2 requires that provenance is signed and hosted by a build service rather than self-generated on a developer's machine. This is where the trust anchor appears. Signing by a build service means the provenance claim carries the identity of the infrastructure that generated it, rather than whoever happened to run the build. That distinction is significant and routinely undervalued.

Level 3 is where provenance becomes structurally resistant to insider threat. The build platform, not the developer, generates the attestation from an isolated, declared process. The developer cannot inject a malicious step after the attestation is generated, because the build platform controls that step entirely. Provenance stops being a gentleman's agreement and becomes an architectural property.

In practice, a Level 3 pipeline looks like this: the build job runs in an isolated environment with declared inputs; a separate provenance job generates an in-toto attestation documenting the entire process; that attestation is signed via Sigstore and pushed to the registry alongside the image. When both jobs complete successfully, the image carries a signed Level 3 provenance attestation that a third party can verify without trusting the team that built it.

Red Hat Konflux, built on Tekton Chains, is a concrete production implementation of this model. It creates an immutable, signed attestation documenting the entire build process and enforces SLSA policies for image reproducibility at the platform level. Attestation generation isn't an optional step bolted onto an existing pipeline; it is a structural property of the build platform itself. Most teams leave open this enforcement gap: they generate attestations in CI but verify nothing downstream before deployment. An attestation that no system ever checks is a substantially weaker guarantee than it appears. The provenance chain is only as strong as its weakest verified link.

Enforcing Provenance at the Deployment Gate with Kubernetes Admission Control

Kubernetes admission webhooks are the mechanism for closing this gap. The Sigstore policy-controller operates as an admission webhook that intercepts pod creation requests before they reach the scheduler and verifies image signatures and attestations against declared policy. The verification happens automatically, at the cluster level, before any pod starts.

A well-formed admission policy can enforce several conditions simultaneously: the image is referenced by digest rather than a mutable tag; a valid SLSA provenance attestation is present and verifiable against a trusted key or Rekor log entry; the attestation asserts a minimum SLSA level; pods failing any of these checks are rejected before scheduling.

The rejection is a hard gate, not a warning. Warnings get acknowledged and deferred. Hard gates enforce unconditionally.

Without admission enforcement, SLSA attestation is a voluntary best practice. Any deployment, from any team, through any mechanism, can bypass it. With admission enforcement set at the cluster level, provenance becomes a hard prerequisite for scheduling. Platform engineers set the policy once; every application team inherits it. This removes the burden of supply chain verification from individual application teams and places it in the infrastructure layer, where it belongs. It also removes the social negotiation problem: no team opts out, because the policy doesn't distinguish by team.

The admission control layer answers one question well: can this image be deployed, given what we know about where it came from? It doesn't answer the adjacent question: what's inside it? That is the domain of the Software Bill of Materials.

SBOMs as the Dependency Layer of Provenance

Modern enterprise applications are composed of roughly 80% third-party code. The average application carries approximately 150 dependencies, 90% of which are indirect, transitive dependencies that no developer explicitly selected. Build provenance tells you where the image came from. It doesn't tell you what's inside it. An SBOM fills that gap with a machine-readable inventory of every component: name, version, license, and origin, tied to a specific, immutable image digest. That digest binding is what makes an SBOM actionable for vulnerability scanning and license compliance, rather than merely descriptive.

Container image layers map naturally to component enumeration. The tooling to generate SBOMs from image layers at build time is mature, and the natural integration point is the build pipeline itself, where the image is produced and its components are known with certainty. Industry commentary consistently notes that containers are where SBOM expectations surface first in procurement requirements, precisely because generation is tractable there.

Docker Hardened Images represent a current reference implementation of the combined model: built to minimize unnecessary components, shipped with complete SBOMs, and carrying SLSA Level 3 provenance attestation. The provenance attestation describes the process; the SBOM describes the contents. Together, they give a downstream consumer, whether a procurement team, a security team, or a regulated customer, a complete and verifiable account of the artifact. Neither document alone is sufficient.

License risk is an underappreciated dimension of the SBOM's value. The 2025 Black Duck OSSRA report found that 56% of scanned codebases contained open-source license conflicts, and 33% contained components with no discernible license. These are not edge cases. They represent legal exposure that compounds with organizational scale and remains invisible without the component-level enumeration an SBOM provides.

Why a Static SBOM Generated Once Is Not Enough

The Omdia 2026 report puts the operational problem in sharp relief: 73% of organizations generating SBOMs report that they enable more efficient vulnerability mitigation, yet more than half generate them only case-by-case rather than at every build. Recognized value and operational practice are separated by a staleness problem most teams haven't fully internalized. That gap is where vulnerabilities go undetected.

An SBOM for a specific image digest remains a valid description of that image indefinitely; the image doesn't change, so the SBOM doesn't become inaccurate. What changes is the threat landscape around the components enumerated in it. New CVEs are discovered against component versions already in a shipped image, sometimes months or years after publication. Without continuous monitoring against the stored SBOM, those vulnerabilities go undetected until someone manually initiates a scan, which happens inconsistently and under pressure.

Continuous monitoring against a stored SBOM enables a materially different operational posture. When a CVE is published against a component version present in a production image, an alert fires automatically. Enrichment with VEX (Vulnerability Exploitability eXchange) data distinguishes exploitable vulnerabilities from theoretical ones, reducing noise without suppressing signal. The audit trail showing when the organization learned of a vulnerability and when it responded is itself a compliance artifact, one that regulators increasingly expect to see.

Only 17% of organizations consistently sign their SBOMs, according to the same Omdia report. This matters because an unsigned SBOM can be altered in transit, quietly, without detection. If the SBOM is the evidentiary basis for a vulnerability response or a license audit, its integrity is mandatory. A signed image attestation establishes chain of custody up to the point where an unsigned SBOM breaks it; the attestation's guarantees don't extend to the artifact's contents unless both are signed.

The architectural direction that makes sense here centralizes container images, SBOMs, and attestations in a single governed private repository, so every artifact and its provenance documents are co-located and versioned together. The fragmented arrangement, where the image lives in one place, the SBOM in another, and the attestation in a third, is where provenance gaps form. It is also more common than the maturity of the current tooling landscape would suggest it should be.

What Bitwise Reproducibility Actually Requires from the Build Environment

Bitwise reproducibility is achievable at ecosystem scale. Large-scale reproducibility experiments have established this empirically, with reproducibility rates ranging from 69% to 91% across a multi-year period spanning continuous package growth. The range itself is instructive: even the best-performing configurations leave a meaningful gap, which means deterministic tooling is necessary but not, by itself, sufficient.

A truly deterministic build environment controls things a Dockerfile doesn't. Every dependency resolves to an exact content-addressed hash, not a version range or a floating tag. The build toolchain is pinned alongside application dependencies rather than assumed from the host environment. No network access is permitted during the build; all inputs are declared in advance and resolved before the build begins. Build outputs are verified against expected hashes before being used as inputs to the next stage. The sequence matters because each control addresses a specific failure mode, and omitting one reintroduces the non-determinism the others were meant to suppress.

The connection to provenance is direct. If the build environment isn't itself reproducible and fully declared, the provenance attestation cannot fully describe what produced the image. An attestation produced by an undeclared, partially floating build environment accurately describes the steps executed, but it can't guarantee that those steps would produce the same artifact again. Verification by reproduction becomes impossible. The attestation degrades from a verifiable claim to an asserted one.

For platform engineers, the practical implication is that the development environment and the CI build environment must be derived from the same declared artifact, rather than separately maintained configurations that happen to resemble each other. When they diverge, two failure modes follow. The first is visible: builds fail in CI that passed locally, disruptive and relatively easy to diagnose. The second is worse. Builds succeed in CI but produce artifacts that differ silently from what the developer tested, and that gap is invisible until something goes wrong in production. The first failure wastes an afternoon. The second one generates a post-mortem.

Closing the Loop Between Development Environment Reproducibility and Image Provenance

The complete chain, stated as artifact lineage, runs as follows. The developer environment is declared, version-controlled, and reproducible, using the same toolchain that CI uses. The build is isolated and hermetic, producing a content-addressed image with an attached SLSA provenance attestation and SBOM. The registry stores the image by digest, with the attestation and SBOM co-located. Deployment is gated by admission control that verifies provenance before the pod starts. SBOM monitoring continues after deployment, alerting against new CVEs published against components already in production.

Most teams aren't there yet. Only 13% of organizations are fully automated in their security check processes; 71% are partially automated or ad-hoc, according to Omdia's 2026 report. Manual gaps break the provenance record. A provenance chain with a manual step in it isn't a chain; it's two segments separated by trust that cannot be independently verified.

The regulatory environment is compressing the timeline whether organizations are ready or not. U.S. Executive Order 14028 and the EU Cyber Resilience Act both mandate SBOM documentation and provenance for software sold into regulated and government markets. According to Omdia's 2026 report, 43% of organizations expect to face an SBOM requirement within 24 months. The organizations building this infrastructure now will satisfy those requirements as a byproduct of sound engineering practice. The ones that wait will build it under deadline pressure, with a higher probability of gaps and less room to fix them.

The threat data validates the investment independently of regulation. Supply chain attacks surged more than 300% since 2024, according to a 2025 Sonatype report. Over 454,600 new malicious packages were identified in 2025 alone, pushing the cumulative known-malware total past 1.233 million packages across major ecosystems. These are active exploitation vectors targeting exactly the gap between what a team believes they deployed and what they actually deployed.

None of this is a security team's project. It is an infrastructure decision that platform engineers make once, and that every application team inherits automatically. The architectural investment is front-loaded. The alternative is reconstructing a provenance chain under incident conditions, working backward through steps that were never fully recorded, trying to determine whether the thing running in production is actually the thing that passed review. That problem does not start at a convenient hour, and it does not resolve cleanly.