Container Image Rebuild Failure Rates in GitHub Actions Workflows

A one-line change to a dependency file, something as small as bumping a patch version, can turn a build that normally finishes in under a minute into a fifteen-minute wait. The code change is trivial. The rebuild is not.

Why container image rebuilds fail

GitHub Actions runners are ephemeral. Every job starts on a freshly provisioned virtual machine that has no memory of any build that ran before it, no matter how recently. That design exists for good reason: isolation between jobs, no contamination from a prior run's leftover state, no risk that one team's secrets or temp files bleed into another team's workflow. But it carries a cost that many teams don't account for until they feel it. Without explicit cache configuration, a docker build step has nothing to work from. It pulls every base image fresh, reinstalls every system package, re-resolves every dependency, and recompiles everything from zero, every single time. A build that takes seconds on a laptop, where Docker's layer cache sits on disk and persists across runs, takes many minutes in CI, where that cache doesn't exist unless someone has gone out of their way to recreate it.

The four categories of failure that cover almost every broken pipeline

Nearly every broken container pipeline falls into one of four categories: infrastructure, configuration, code, and integration. Sorting a failure into the right bucket early cuts diagnostic time substantially, because each category points toward a different kind of fix and often a different owner on the team. The discipline that matters most here is tracing an error back to where it actually originates rather than where it happens to surface. A test failure late in a pipeline might really trace back to an uncached dependency installed hours earlier in a different job. A deployment timeout might really be a misconfigured health check that has nothing to do with the deployment step itself. For container image rebuilds specifically, the weight falls overwhelmingly on infrastructure and configuration. The code being built is rarely the problem. What breaks is almost always the scaffolding around it: how the cache is keyed, which architecture the image targets, what permissions the workflow token carries, and whether the platform running the job is behaving the way it's supposed to. Treating these four categories as one undifferentiated mess of "the pipeline broke" is what wastes debugging cycles. Treating them as distinct, each with its own typical cause and typical fix, is what gets a broken build back to green quickly. The sections that follow take each of the major failure modes in turn, starting with the one most within a team's direct control.

Cache key design: the highest-leverage local failure mode

A cache that never hits is, in the overwhelming majority of cases, a cache that was keyed wrong. The most common pattern is a key that changes on every single run, whether or not anything about the dependencies actually changed, which guarantees a cold cache every time and defeats the entire point of caching. The fix is a stable key derived from a lockfile rather than from a hash or timestamp that shifts with each commit. A key built using something like hashFiles('**/package-lock.json'), or the equivalent lockfile for whatever package manager a project uses, changes only when the actual dependency graph changes. It stays fixed across commits that touch application code but leave dependencies alone. The cache keeps paying off run after run instead of resetting itself for no reason. The opposite pattern, an unstable key built from an unpinned hash, a timestamp, or any value that shifts per run, guarantees a cold cache on every single execution. It looks like caching is configured. It behaves as if no caching exists.

Some teams push back on this, arguing that caching introduces a risk of building against stale dependencies. The concern is reasonable on its face, but it misunderstands how a lockfile-keyed cache actually behaves. Pinning the cache key to the lockfile means the cache invalidates at precisely the moment the dependencies change, which is exactly the moment a fresh pull is warranted. The cache defers the expensive work until the one point where that work is actually necessary, and skips it everywhere else.

Architecture mismatch as a silent rebuild killer

Architecture mismatch is dangerous precisely because it doesn't announce itself. A build can succeed on a developer's machine, succeed again in CI, and still produce two different images that behave differently once deployed, all without a single error message pointing to the cause. The mechanism is straightforward once it's named: a base image tag like node:latest or postgres:15-alpine doesn't point to one fixed image. It resolves to a different digest depending on the architecture requesting it. A developer running an ARM-based machine pulls an ARM image under that tag. A GitHub Actions runner running on amd64 pulls an x86 image under that exact same tag. Each build succeeds on its own terms, independently, with no error. The divergence appears later, in behavior differences between what was tested locally and what runs in production.

The deploy workflow for the vib project illustrates the fix directly: its GitHub Actions build pins explicitly to linux/amd64, a deliberate decision made to match the production target and to prevent exactly this kind of silent drift between what gets built and what gets deployed. The general version of the fix is to pass --platform linux/amd64 (or whatever the actual target platform is) explicitly to the build step, or to use a matrix strategy when a project genuinely needs to support multiple target architectures. Either approach replaces an implicit assumption with an explicit declaration. This is really a specific case of a broader problem: when a build environment isn't declared explicitly, the platform fills in the gaps with defaults, and those defaults have no obligation to match what production actually runs.

GITHUB_TOKEN permissions and registry authentication as a configuration trap

A container image can build correctly, pass every test, and still fail at the very last step, when the workflow tries to push it to a registry. The build itself isn't the problem. The workflow's token lacks the explicit write permissions that GitHub now requires by default. A build job that completes cleanly can die at the push step with a permissions error that has nothing to do with the image itself.

The fix is to declare permissions explicitly in the workflow's YAML rather than relying on defaults: packages: write to push to the GitHub Container Registry, contents: write for repository access, and id-token: write where a job uses token-based authentication to reach a cloud provider. A related trap catches teams even after they've set permissions correctly: secrets scoped to a specific environment, such as a production environment, are invisible to any job that doesn't explicitly declare environment: production in its own configuration. The secret lookup silently returns empty in that case, and the registry push then fails with a credential error that reads as an authentication problem rather than what it actually is, a scoping problem.

The deploy workflow for the vib project models the pattern that avoids both traps. Each job is granted only the permissions it actually needs, actions referenced in the workflow are pinned by commit SHA rather than by a mutable version tag, and the GHCR authentication step uses the job's own GITHUB_TOKEN scoped to packages:read for the pull step specifically. Pinning by SHA rather than tag can look like unnecessary caution to a team that just wants a workflow to run. But a tag like @v3 is mutable. Whoever maintains that action can update what @v3 points to without notice, and a workflow that looked stable yesterday can change behavior today with no change made on the consuming side.

Platform-level instability as an infrastructure failure teams cannot configure their way out of

Not every rebuild failure traces back to something a team got wrong. GitHub Actions itself has gone through a documented pattern of service disruptions serious enough to halt workflows in the middle of a run, and the scale of that pattern is now well established. On August 6, 2026, GitHub Actions experienced degraded availability severe enough that some workflow runs failed to start, others failed partway through execution, and some calls to the Actions REST API returned errors. Engineers identified the source of the problem and began mitigation, but full restoration took hours.

That event wasn't an isolated incident. A tracking study covering the period from May 2025 to April 2026 recorded 257 incidents in total, 48 of which qualified as major outages, with overall availability sliding steadily across the period and dropping sharply in early 2026. Research published in Future Internet examined what made the platform susceptible to failures of this kind, pointing to three recurring structural weaknesses: services coupled tightly enough that a small, localized fault could cascade outward, weak protection against anomalous or misbehaving client traffic, and capacity that couldn't expand fast enough to absorb demand spikes that fell outside normal, predictable usage cycles. AI-driven demand acted as both trigger and amplifier for that third weakness. A February 2026 incident showed what cascading failure actually looks like in practice: a gap in telemetry data triggered the blanket application of security policies across every region at once, which blocked metadata access for the virtual machines running hosted jobs, and the effect was simultaneous across all regions rather than contained to one.

The practical cost to a container build is concrete. A workflow in the middle of a multi-minute image build can fail partway through if the runner loses connectivity to the cache backend or to the registry API mid-job. When that happens, the build doesn't resume from where it left off. It restarts from scratch on the next attempt, and every intermediate layer built up to that point is simply gone. Some teams have responded by standing up self-hosted runners or hybrid CI setups that reduce how much they depend on a single vendor's infrastructure for every build. That response comes with its own cost: self-hosted infrastructure needs patching, scaling, and monitoring that a fully managed runner fleet would otherwise handle.

How rebuild failures compound into supply chain integrity gaps

An uncontrolled rebuild isn't just a delay. It produces a new image digest, and any cryptographic signature or attestation tied to the digest that preceded it stops being valid the moment that new digest exists. A rebuild failure, viewed this way, is also a supply chain integrity failure. Cosign verification breaks in production for exactly this reason: an image gets rebuilt after it was already signed, the rebuild produces a different digest than the one that was signed, and the original signature no longer matches anything that actually exists in the registry. The fix is to pass the exact digest produced by the build job's own output into every downstream step, rather than referencing a mutable tag, and to make sure no re-tagging happens anywhere between the signing step and the verification step.

The deploy workflow for the vib project reflects this discipline directly. Images are tagged with the commit SHA that produced them, never with a mutable tag like latest, so the digest that was built, the digest that was pushed, and the digest that was pulled at deploy time are guaranteed to be one and the same digest that was actually reviewed. Modern container hardening practice builds on that same foundation: automatic SBOM generation in CycloneDX or SPDX format on every build, cryptographic signatures applied through Sigstore's cosign, and SLSA provenance attestations, all of them tied to one specific build and its one specific digest. The moment a rebuild happens outside that controlled workflow, whether because a cache miss forced an unplanned fresh build or because a platform outage restarted a job mid-run, the resulting image can carry a brand-new digest with no SBOM and no attestation attached to it anywhere. That's a provenance gap, and it's where the operational failures described earlier in this piece and the security failures described here turn out to be the same failure viewed from two angles. Fixing rebuild reliability is a precondition for any claim a team wants to make about the trustworthiness of its software supply chain.

Immutable artifacts and declarative environments

The teams that stop having rebuild problems are the ones who've arranged their pipelines so that the same artifact, built exactly once, gets promoted unchanged through every later stage, with no opportunity for a second build to sneak in anywhere along the way. Mature CI/CD practice treats artifacts as immutable on principle: nothing gets rebuilt between staging and production, full stop, and whatever artifact passed testing is the same artifact that gets deployed.

The architecture behind the vib project encodes that principle structurally, as a property of the system rather than as a policy someone has to remember to follow. GitHub Actions builds the image exactly once, pushes it to GHCR tagged by commit SHA, and the server does nothing but pull that already-built image. Build and deploy are kept as structurally separate steps, which makes a deploy-time rebuild impossible rather than merely discouraged. Manual, undeclared environment configuration is the upstream cause behind most ad-hoc rebuilds in the first place: when a build environment isn't pinned and declared explicitly, small differences between one run and the next produce different images from what looks like the same source, and teams end up rebuilding just to chase a "clean" result. The rebuild, in that light, is a symptom of an environment that was never made deterministic to begin with.

Declarative environments solve this at the root. When a full dependency graph is pinned and reproducible, a build run on a developer's laptop, in CI, and on a production runner all produce the identical result, and the "works here, fails there" condition that drives teams to rebuild in the first place simply has no way to occur. Tools like Nix provide one well-established foundation for this kind of guarantee: environments are defined declaratively, and the same inputs resolve to the same artifact every time, regardless of where the build runs. Building on that foundation gives teams access to reproducible, pinned environments without requiring every engineer to learn Nix directly, and it enables hash-level tracking that ties each deployment back to its exact set of inputs. Rebuild failures stop happening because the environment itself no longer has room to drift.