Evaluating Software Supply Chain Security Tools

The market offers seven broadly recognized tool categories: secure-by-default infrastructure, software composition analysis (SCA), secret and credential scanning, CI/CD pipeline security, container image hardening, SBOM and provenance generation, and infrastructure-as-code (IaC) security. Vendors routinely claim coverage across several of these simultaneously, which makes category-based comparison charts structurally misleading. When a vendor claims both SCA and CI/CD pipeline security, the chart puts a checkmark in two columns, but neither checkmark tells you whether either capability is adequate at the stage where your specific risk actually enters.

Two incidents make this concrete. In August 2025, a compromised maintainer token on the Nx build system enabled a malicious npm package that executed on install, stole credentials, and attempted to abuse locally installed AI CLI tools for reconnaissance. Seven months earlier, StepSecurity disclosed that historical Git tags on a popular GitHub Action used by more than 23,000 repositories had been modified, exfiltrating CI/CD secrets to public logs. These threats entered at entirely different points: a package registry and a CI runner. Separate tools address each. I've sat through vendor briefings where a single platform claimed to cover both, and the honest answer, when you press on specifics, is almost always that one of the two is a checkbox and not a capability.

Veracode's 2026 State of Software Security Report recommends evaluating tools on five capability dimensions: Detection, Prevention, Remediation, Integration, and Compliance. These are the right questions. But they're not a deployment map. They tell you how well a tool does what it does; they don't tell you where to aim it. The organizing principle that actually drives a coherent evaluation is the stage at which a threat enters — dependencies and open-source components, build pipelines and CI/CD infrastructure, developer environments and local tooling, and artifact provenance and delivery. Buyers who start with stages rather than categories can identify gaps in their own posture before they talk to a single vendor.

Evaluating tools at the dependency ingestion stage

SCA tools operate here, scanning declared and transitive dependencies for known vulnerabilities, license issues, and malicious packages. The first evaluation question is whether the tool surfaces transitive dependencies, not just direct ones. An application declares a modest number of direct dependencies while pulling in hundreds of transitive packages that no engineer ever explicitly chose. A tool that reports only on declared dependencies is reporting on a fraction of actual exposure, and practitioners frequently don't realize this until they're trying to scope an incident.

The second question is whether the tool detects malicious packages, including typosquatting and dependency confusion attacks, or only CVE-matched vulnerabilities. These are genuinely different detection problems. CVE matching is a database lookup. Detecting a package designed to impersonate a legitimate one requires behavioral and structural analysis. Many tools don't perform that analysis, and their marketing materials rarely say so plainly.

Third, how quickly does the tool detect a newly published malicious version? The Nx incident demonstrated that a compromised package can execute and cause damage within the install window, before any CVE has been assigned and before most databases have been updated. Speed of detection at the ingestion boundary is a separate evaluation criterion from breadth of coverage. Conflating them produces dangerous blind spots.

Secret and credential scanning belongs at this stage as well. Leaked tokens embedded in dependencies or build manifests are a dependency-stage risk, not only a source code risk. The integration criterion matters acutely here. Tools that sit outside the developer's normal workflow — outside the IDE, the package manager hook, the pull request check — will be bypassed under deadline pressure. That's not a behavioral problem unique to any team; it's a predictable outcome of friction. The honest probe at this stage is whether the tool flags behavioral risk, such as packages that execute code on install as the Nx payload did, or whether it limits itself to version-matched CVEs. Those aren't the same capability, and no amount of marketing language makes them equivalent.

Evaluating tools at the build pipeline stage

SolarWinds SUNBURST is the canonical illustration of why build pipeline security is a distinct and non-substitutable stage. Every software component in the affected build was legitimate. The attacker compromised the build pipeline itself and injected malicious code during compilation, which was then packaged into the output. An SBOM of that build would have listed clean, trusted components and missed the breach entirely. This is the conceptual boundary between dependency scanning and build integrity tooling — SCA looks at what goes in; build security evaluates whether the process itself was tampered with. Failing to understand that boundary is how organizations end up with a thorough dependency program and a completely unguarded build.

SLSA, the Supply-chain Levels for Software Artifacts framework, is the current standard for build provenance. Version 1.2, released in November 2025, defines a Build Track with three levels of increasing integrity assurance and introduces a Source Track that extends provenance upstream to the source code itself. Provenance attestations create a cryptographic trail that enables identification of compromised builds and supports rollback decisions.

The evaluation questions follow directly. Does the tool generate signed, verifiable provenance on every build, not only on release builds? An organization that attests release artifacts but not intermediate builds has a meaningful gap in its audit trail. Does it isolate build processes so a compromised runner can't affect parallel builds or exfiltrate secrets to adjacent jobs? Can it detect modification of CI/CD configuration files or runner environments between runs?

The implementation reality is that modern CI/CD pipelines frequently lack clear identity boundaries. Scaling SLSA verification across diverse runner environments and artifact flows is a genuine operational challenge, not a problem that a procurement decision alone resolves. The OWASP CI/CD Top 10 and NIST SP 800-204D provide complementary checklists for evaluating pipeline-specific controls. Tools at this stage that produce signed attestations also carry compliance value beyond security — those logs directly satisfy NIST SSDF and ISO/IEC 27001 audit requirements, which matters when procurement decisions need organizational justification beyond the security team.

Evaluating tools at the developer environment stage

The Nx incident moved the attack surface explicitly onto developer machines. The payload didn't stop at credential theft; it attempted to abuse locally installed AI CLI tools for reconnaissance. The threat model now includes workstations. This is not a theoretical expansion of scope.

The underlying condition that makes this stage difficult to secure is environment drift, and the numbers here are arresting. In a study of 5,298 Docker builds, Malka et al. (January 2026) found that only 6.4% of rebuilt images matched the original installed package versions exactly. The "standardized" developer environment most teams believe they have is, in the majority of cases, a fiction. The drift consequences for security are direct — when different developers run different tool versions, a malicious package or tampered binary affects some environments and not others, which complicates both detection and incident scoping. It also breaks the assumption that a clean CI build implies clean developer builds — an assumption I've seen security teams carry unchallenged for years.

The evaluation questions at this stage probe whether tooling enforces a declared, version-controlled environment or merely recommends one. There's a material difference. Recommendation-based approaches rely on human compliance; enforcement-based approaches make deviation the exception. Beyond that — is the environment definition cryptographically tied to its inputs, so that substitution is detectable? Does the tooling work consistently across architectures and operating systems, because the cross-platform surface area is itself part of the attack surface?

Cryptographic-hash-based approaches to environment management address this structurally. When every package is identified by a hash of its inputs, including source, dependencies, build flags, and compiler version, identical inputs always produce the same package path. This enables reproducible builds, though it doesn't automatically guarantee them in all downstream workflows. The Enduring Security Framework working group has specifically identified reproducible builds as a key defensive technology.

Containers narrow this gap but don't close it. The same Dockerfile built two weeks apart via a standard package manager update produces different outputs, because nondeterministic build steps are among the principal causes of the mismatch rate documented in the Malka study. Tools to evaluate at this stage include environment managers capable of pinning dependencies all the way to system libraries, not only language-level packages, and IaC diff tooling that detects divergence between declared and actual states.

Evaluating tools at the artifact provenance and delivery stage

This stage covers everything that leaves the build and reaches a consumer: container images, binaries, published packages, and the metadata traveling with them. SBOMs are the primary deliverable. A software bill of materials is a catalog of all components, including libraries, dependencies, and modules, with version numbers, licensing data, and provenance. The two leading formats are CycloneDX and SPDX; tools should generate and validate both on every build, not only on demand.

CISA updated its Minimum Elements for an SBOM in August 2025, replacing the 2021 NTIA document. The updated guidance replaces "Supplier Name" with "Producer Name" and adds support for cryptographic hash inclusion per component, enabling software component attestation. Any tool that was compliant with the 2021 NTIA standard requires re-examination against the 2025 CISA update. That's not a minor revision; it changes what completeness means.

The operational value of SBOMs is clearest under incident conditions. When Log4Shell emerged, organizations with current SBOMs identified affected applications within minutes; those without them spent days or weeks. That speed difference directly determines the size of the exposure window, which is the actual risk metric. Gartner projected that 60% of organizations building or procuring critical infrastructure software would mandate SBOMs by 2025. Teams that build generation into their workflows now won't be scrambling when contract language requires it.

The evaluation questions at this stage — does the tool generate SBOMs automatically on every build, or only when someone manually triggers the process? Does it capture transitive dependencies, or only direct ones? SCA tooling often provides more complete transitive coverage than SBOM generators alone, which creates an argument for integrating both at this stage rather than treating them as alternatives. Are outputs cryptographically signed and machine-consumable for automated downstream workflows, or are they human-readable PDFs that require manual handling?

I'll state a limitation plainly here, because the practitioner community has been reluctant to. Most organizations provide SBOMs only upon request, and conversations at the Government Secure Supply Chain Summit in July 2025 surfaced serious questions about whether tool outputs could be trusted and whether the generation processes themselves were secure. SBOM quality varies significantly across tools. A procurement decision doesn't resolve this; it only begins the work.

SBOM and build provenance are complementary, not interchangeable. A clean SBOM says nothing about pipeline integrity, as SolarWinds demonstrated conclusively. A provenance attestation says nothing about whether a listed dependency carries a known CVE. Both are necessary, and they address different questions.

How to use the five capability dimensions to score tools within each stage

Once stages are mapped to an organization's actual risk posture, Veracode's five dimensions — Detection, Prevention, Remediation, Integration, and Compliance — become a scoring rubric applied within each stage, not a single evaluation pass across the entire market. A tool that scores well on Compliance at the artifact stage scores poorly on Integration at the build stage. Conflating those scores produces a false aggregate, and I've watched that conflation justify purchases that left real gaps unaddressed.

What each dimension means in practice, applied per stage — Detection asks whether the tool catches threats at this specific entry point, including behavioral anomalies in packages, tampered runner configurations, drifted environment hashes, and unsigned artifacts. Prevention asks whether it blocks or gates — refusing to install a flagged package, failing a build without a valid attestation, enforcing a locked environment definition. Remediation asks whether it surfaces actionable next steps rather than just alerts: pinned upgrade paths, rollback-capable provenance logs, environment diff reports a developer can act on within their normal workflow. Integration asks whether the tool fits into the stage's natural workflow — a package manager hook, a pull request check, a CI step, an environment activation command — or whether it requires a parallel process that developers will eventually route around. Compliance asks whether it produces artifacts satisfying EO 14028, FedRAMP Rev. 5, NIST SSDF, or ISO/IEC 27001 for this stage specifically, not generically.

The most common scoring failure mode is a tool that performs well on Detection and Compliance but poorly on Integration. The alerts exist. The compliance boxes are checked. But developers have found a path around the friction, and the control is providing theater rather than protection. Tools that retroactively scan finished builds can't satisfy continuous monitoring controls on their own. Applying the Integration dimension at the build and environment stages surfaces that gap precisely, and surfaces it before a contract is signed rather than after.

The scoring exercise also distinguishes genuine multi-stage tools from vendor overclaiming. Applied stage by stage, the question isn't whether a vendor claims to cover a category; it's whether the tool's specific capability addresses the threat at this specific entry point. That becomes concrete and defensible in a way that category checkboxes never are.

What a coverage map across all four stages reveals about typical tool stacks

Most mature security programs have reasonable coverage at the dependency stage. SCA is the oldest and most broadly adopted category, and its presence in most tool inventories reflects that history. Coverage becomes progressively thinner moving through the remaining three stages.

The build stage is frequently the weakest. Complex, highly automated pipelines often lack the clear identity boundaries that SLSA verification requires. Scaling build provenance across diverse runner environments is an implementation challenge that most organizations haven't yet resolved, and many haven't started. The developer environment stage is the most commonly absent from tool inventories entirely, which is striking given what the Nx incident demonstrated. Attacks targeting developer tooling are active and documented. This is not a hypothetical threat category.

The artifact and provenance stage has regulatory momentum from EO 14028 driving SBOM adoption, but the operational gaps remain significant. Automated generation on every build, reliable transitive coverage, and machine-consumable signed outputs aren't yet the norm. The Government Secure Supply Chain Summit conversations in 2025 confirmed that practitioners are still working through basic questions of trust and tooling quality. Regulatory pressure and operational readiness are not the same thing.

AI development pipelines represent an emerging gap that cuts across all four stages. ReversingLabs' 2026 report identified active efforts to infiltrate AI development pipelines. AI coding agents interact with dependencies, build environments, and local tooling in ways that existing coverage assumptions weren't designed to address. The surface is expanding faster than the evaluation frameworks most teams use.

A coverage map also surfaces overlap — two tools claiming the same stage vector, duplicating cost and effort without reducing risk. That's frequently where consolidation decisions become defensible, not because of vendor preference, but because the map makes redundancy visible and quantifiable.

The coverage map is also a communication artifact. It translates security posture into stage-by-stage risk exposure that engineering leadership and procurement can engage with directly, without requiring deep security expertise from either audience. Security frameworks that only practitioners can interpret don't drive the purchasing decisions and organizational commitments that supply chain security actually requires. Getting that translation right is as much a part of the work as getting the tool selection right.