Container Security Best Practices for Production Kubernetes Workloads

The CNCF's annual survey found that container adoption in production climbed substantially between 2023 and 2025, and 82% of those shops run Kubernetes to manage the fleet. That growth has made Kubernetes clusters a genuinely attractive target, and the platform's out-of-the-box configuration does attackers a favor: pods can run as root by default, there's no network segmentation between pods unless someone adds it, secrets sit base64-encoded rather than encrypted, and default service accounts carry more permission than almost any workload needs. The CIS Kubernetes Benchmark runs past 200 recommendations, which tells you hardening is a discipline you keep practicing, not a checkbox you tick once. It's a discipline you keep practicing.

Three ideas hold the whole thing together, and Convotis's Kubernetes security guidance frames them well: defense in depth (controls at every layer, not just a perimeter firewall and a prayer), least privilege (users, services, and containers get exactly what they need and nothing extra), and zero trust (no request gets a free pass just because it came from inside the cluster). The rest of this piece walks through five control points, image integrity, supply chain provenance, control plane and RBAC, network segmentation, and secrets management, because a weak link in any single one can undo the work done everywhere else.

How attackers move through a Kubernetes environment

An industry data breach investigations report found that vulnerability exploitation climbed to 31% of breaches as the single most common way attackers get in. That figure puts a number on something practitioners already assumed: an unpatched base image or a stale dependency is usually where the trouble starts, not some exotic zero-day nobody could have caught.

From there the path is fairly predictable. Code execution happens inside a container, usually through an exploited CVE in a base image or a library nobody rebuilt in months. The attacker then moves sideways, helped along by pod-to-pod networking that has no restrictions and service accounts that carry more permission than the workload asked for. If the container is privileged, mounts a host path, or a kernel exploit is available, the attacker escapes the container entirely and lands on the host. From the host, a compromised image can get pushed back into a registry and pulled by every environment that trusts it, turning one bad build into many.

Timing makes the whole thing worse. Around 70% of containers live five minutes or less, so by the time anyone notices something's wrong, the container that did the damage is long gone, and snapshot-based scanners built for longer-lived infrastructure often never see it. A related failure mode at scale: when a running container can't be traced back to the exact commit, pipeline, and security check that approved it, there's no way to actually measure risk. The container is out there doing something, and nobody can say with confidence how it got there.

Each of these stages, code execution, lateral movement, host escape, image distribution, maps to a different layer of control. No single tool or vendor closes off the whole chain. That's the point of walking through it stage by stage: the fix for one link doesn't cover the others.

Building images that are hard to exploit before they leave the build stage

A vulnerable base image doesn't cause one problem. It causes as many problems as there are pods pulling that image, and The multiplication effect is straightforward: one risky base image spreads its risk across every service built on top of it.

Start with what's actually in the image. Distroless or scratch-based images strip out the shell, package manager, and network utilities an attacker would otherwise use once they've got code execution, so there's simply less to work with after a compromise. ox.security's container security guidance suggests flagging anything older than 90 days for an automatic rebuild, since a stale image is quietly accumulating CVEs nobody's looked at. And image tags should point to a specific digest, not a mutable tag like latest, because a mutable tag means the image that passed your scan isn't necessarily the image that actually gets deployed. Someone could push a new image under the same tag an hour later, and the deployment pulls it without complaint.

Scanning has to happen in CI, before an image ever reaches the registry. Trivy and Grype are the tools most teams reach for. But scanning every CVE with equal urgency wastes time on things that may never get exploited, so it helps to filter against CISA's Known Exploited Vulnerabilities list and fix what's actively being used in the wild first, ahead of anything that's merely high-severity on paper. CI scanning catches what's known at build time; runtime scanning is what catches a CVE disclosed after the image already shipped, which is a different job and needs a different tool.

A secret baked into an image layer survives even after someone deletes it from the Dockerfile, because the layer history still holds it. Manifests, Helm charts, and Kustomize templates all need the same scrutiny for misconfigurations, privileged containers, wide-open network access, missing security contexts, since an insecure default buried in a template can slide past a manual review that only checks the final rendered configuration file.

Signing closes the loop. Cosign, part of the Sigstore project, lets a runtime verify an image hasn't been tampered with between the build and the deployment. That signature is also the prerequisite for the admission-control enforcement covered in the next section: an image with no signature can't be verified at the gate, full stop.

A few pod-level settings belong in this stage too, even though they take effect at deploy time: runAsNonRoot: true, privilege escalation disabled, readOnlyRootFilesystem turned on wherever the app allows it, and Linux capabilities dropped down to whatever the workload actually uses. None of these are exotic. They're just easy to skip under deadline pressure.

Enforcing software supply chain integrity from source commit to running pod

Reporting on dev.to by saaro_net projects that supply chain attacks will cost the global economy $80.6 billion in 2026, a 76% jump from 2023. The Jaguar Land Rover incident in late August and early September of 2025 caused £1.9 billion in damage and five weeks with production halted. The Jaguar Land Rover incident is a car company that stopped making cars. That's a car company that stopped making cars.

Open-source vulnerabilities have grown substantially in recent reporting cycles, and part of the driver is AI-generated code pulling in transitive dependencies nobody actually reviewed. Regulation has caught up to the risk: EU Cyber Resilience Act reporting obligations take effect in September 2026. SBOMs and verifiable build integrity stop being a nice-to-have and become a legal requirement for a lot of organizations.

An SBOM, or software bill of materials, is a machine-readable list of every component in an application and its runtime, proprietary code, open-source libraries, container images, binaries, and every transitive dependency they depend on. CISA now expects these in SPDX or CycloneDX format specifically so they can be generated and consumed automatically inside a pipeline rather than assembled by hand after the fact. Even so, a real gap persists: plenty of mature organizations still produce incomplete SBOMs, in large part because many open-source projects have never generated supply chain artifacts for their own code. You can't inventory what upstream never documented.

An SBOM tells you what's there. Admission enforcement is the job of stopping things from running, and the two need to work together. That's the job of admission enforcement, and the two need to work together.

Stribog.com's writeup describes the trust chain that's forming as the standard in 2026: a source commit triggers a build that produces SLSA Level 3 provenance, typically through the slsa-github-generator tooling, designed to produce provenance that the build system itself cannot tamper with. That provenance gets attached to a Cosign keyless signature, logged in the Rekor transparency log, and checked by a Kyverno admission policy that blocks anything unverified before it ever reaches a node. SLSA Level 3 is increasingly treated as the bar production workloads should be clearing in 2026, not an aspirational target for later.

Admission control is what makes every step upstream of it actually enforceable. Without Kyverno or OPA/Gatekeeper sitting at the gate, signing and provenance attestations are just paperwork. Nice paperwork, maybe, but nothing stops an unsigned image from running if nothing's checking.

And confidence is running ahead of practice here. DigiCert's State of Software Supply Chain Security report found close to half of organizations rate their own programs as "established" or "optimizing," yet only 13% have automated the relevant controls. That's a wide gap between how safe people feel and how safe they've actually made things.

Environments built through reproducible, declarative definitions, where every package version and every build input is pinned explicitly rather than resolved at build time, tend to produce more accurate SBOMs as a byproduct of how they're built, not as a separate scan bolted on afterward.

Locking down the control plane and RBAC before any workload runs

A compromised control plane doesn't threaten one workload. A compromised control plane threatens every workload the cluster runs. That is why lateral movement inside Kubernetes is so much worse than lateral movement in a flat network of individual servers.

The API server is the front door, so it needs a private endpoint, firewall rules, and IP allowlisting rather than open exposure to the internet. Every connection to it should run over TLS. And audit logging needs to be turned on and shipped somewhere central, because logs sitting on the API server node itself don't help much once that node is the thing under investigation.

etcd deserves particular attention since it holds all cluster state, including every secret, even the base64-encoded ones. Access should be restricted to the API server exclusively, and encryption at rest should be turned on, because an unencrypted etcd store means anyone who gets at the underlying storage can read every secret in the cluster without needing to touch Kubernetes' RBAC at all.

RBAC is where a lot of clusters quietly go wrong. Wildcard roles, the kind that grant access to everything rather than a scoped list of resources, are a lateral movement path waiting to be used. Service accounts should be scoped tightly to the specific workload that needs them, and automountServiceAccountToken should be disabled on any pod that has no business calling the API server at all. An OIDC connection to an external identity provider, or workload identity federation through a cloud provider's IAM, issues short-lived credentials instead of a token that sits valid for months, and long-lived static tokens should be replaced wherever possible.

Pod Security Standards replaced the older PodSecurityPolicy, which Kubernetes removed as of version 1.25. The replacement gives three profiles: Privileged, which is effectively unrestricted, Baseline, which blocks the well-known privilege escalation paths, and Restricted, which is full hardening meant for application workloads. Restricted should be the default enforced through Pod Security Admission, and a policy engine like Kyverno or OPA/Gatekeeper can layer custom rules on top of that baseline for anything specific to the organization. Multi-factor authentication on administrative access rounds this out: it's a cheap, unglamorous control, and it still meaningfully reduces credential-based compromise.

Network segmentation: moving from fully open to default-deny

Out of the box, Kubernetes networking has no restrictions at all. Out of the box, Kubernetes networking has no restrictions at all, and every pod can talk to every other pod in the cluster, so a single compromised container starts with an internal attack surface as wide as the entire cluster.

The fix is a default-deny NetworkPolicy in every namespace, with explicit allow rules added for whatever traffic genuinely needs to flow. It has to cover both ingress and egress. Teams frequently lock down inbound traffic and leave egress wide open, which quietly preserves an outbound channel an attacker can use to talk to command-and-control infrastructure even after the "obvious" holes are closed. Namespace-level policy also does real work limiting blast radius: a pod compromised in one namespace shouldn't be able to reach directly into workloads sitting in another.

A service mesh like Istio adds encryption between services through mutual TLS, and it hands out a workload identity that authorization decisions can key off of. A network policy only decides whether traffic is allowed to arrive. mTLS decides whether the thing that arrived is actually who it claims to be, which is a separate question and a separate layer of protection.

At the edge, a web application firewall and rate limiting on the ingress controller cut down the attack surface visible from outside the cluster, and TLS termination and certificate management at that layer should run on automation rather than someone's manual renewal calendar.

None of this holds if a privileged container can just bypass the network layer entirely at the host level. hostNetwork: true and host path mounts can circumvent the isolation that network policies are meant to provide, which is exactly why the pod hardening settings from the image-building and control-plane sections aren't optional extras. They're the thing that makes network segmentation mean anything in practice.

Secrets management: getting credentials out of the cluster and into runtime injection

Kubernetes secrets are base64-encoded, and base64 is encoding. Anyone with RBAC read access to secrets in a given namespace can trivially decode them, since base64 encoding provides no actual protection.

The production-grade fix is to keep secrets outside the cluster entirely and inject them at runtime instead of storing them as Kubernetes objects. HashiCorp Vault is the tool most widely deployed for this, and it gets into pods either through the Vault Agent Sidecar Injector or the CSI Secrets Store Driver, neither of which requires rewriting the application to fetch its own credentials. Cloud-native alternatives, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, all follow the same injection pattern through that same CSI driver.

For clusters not yet ready to move secrets management fully outside Kubernetes, a few native changes still cut real risk. Encryption at rest on etcd protects secrets even if the underlying storage gets compromised. Mounting secrets as files rather than environment variables cuts real risk, since environment variables tend to leak through application logs, error dumps, and child processes that inherit the parent's environment without anyone intending it. And disabling automatic service account token mounting on pods that don't call the API removes a credential that was never needed in the first place.

Rotation is where external secrets managers pull clearly ahead of native Kubernetes secrets, which offer no built-in rotation, versioning, or audit trail on their own. A secret pulled through a CSI driver can rotate without restarting the workload that consumes it, which native Kubernetes secrets simply can't do without extra tooling bolted on. That gap, rotation without a restart, is a small detail on paper. Operationally, a credential leak either gets fixed in minutes or lingers until someone finally notices, depending on rotation without a restart.