Dependency Provenance Testing Is the Release Gate Most Pipelines Skip

A mid-sized SaaS company was three weeks into a due-diligence process with an acquirer's engineering team when a reviewer asked a question nobody on the engineering side had prepared for: "Can you produce a bill of materials for the build you shipped to production most recently, and show us how you verified that every package in it is what it claims to be?"

The engineering leader answering that question had good numbers to offer. Ninety-four percent unit test coverage. A pull request policy that required two approvals and a passing static analysis gate before merge. A changelog of every code review comment for the past two years. What she did not have was an answer to the actual question. Her team could tell you, with confidence, that the code her engineers had written did what it was supposed to do. She could not tell you, with the same confidence, that the express, lodash, requests, or any of the other four hundred-plus open-source packages compiled into that most recent build were the packages their maintainers had actually published, unaltered, from an identifiable source. Nobody on her team had ever been asked to verify that. Nobody's job was to verify that. The pipeline had never been built to check it.

This is not a story about negligence. It's a description of how almost every software organization operates today, including many with mature QA practices and strong engineering discipline. The reason is structural: code review, unit testing, static analysis, and QA test plans were all designed to answer one question — does the code we wrote behave correctly? They were never designed to answer a different question — is the code we didn't write actually the code we think it is? Those are different verification problems, and the industry has spent two decades building excellent tooling for the first one while leaving the second one almost entirely unaddressed in most release pipelines.

Dependency provenance — the verifiable chain of custody connecting a package sitting in your build to the source code and maintainer identity it claims to come from — is a testable release property. It can be checked automatically, gated in CI/CD, measured over time, and reported on the same dashboards that already track test coverage and deployment frequency. Most pipelines don't check it, not because the tooling doesn't exist, but because almost nobody has framed it as a release-engineering and quality assurance responsibility rather than a periodic security audit exercise. This article makes that case, walks through the mechanics of doing it, and gives engineering leaders a way to assess where their own pipeline stands.

The Verification Gap Your Test Suite Was Never Built to Close

Every serious engineering organization has some version of a quality gate: code review, automated tests, static analysis, sometimes dynamic analysis or fuzzing. These controls share a common assumption that is so basic it rarely gets stated out loud — they assume the code under test was written with intent you can evaluate. A reviewer reads a diff and asks whether the logic matches the ticket. A test asserts that a function returns the right value for a given input. A static analyzer flags a pattern that historically correlates with bugs. All of this machinery operates on code your team is responsible for and, in principle, understands.

None of it operates on the code you didn't write. When your build pulls in a package from npm, PyPI, RubyGems, Maven Central, or any other registry, that package usually arrives as a compiled or minified artifact, or as source you will never manually review, sitting several levels deep in a dependency tree you didn't design. A 2023 analysis published by the Open Source Security Foundation, drawing on data reported through GitHub's Octoverse research, found that a typical JavaScript project with ten direct dependencies pulls in an average of 683 indirect (transitive) dependencies once the full tree resolves. A separate 2020 academic study cited in the same analysis found the average JavaScript program depends on 377 packages overall, and that roughly ten percent of JavaScript packages depend on more than 1,400 third-party libraries. Across ecosystems, publicly available industry estimates commonly put the open-source share of a typical modern application's codebase somewhere between 70 and 90 percent.

That means the overwhelming majority of the code running in production, for a typical company, was not written or reviewed by anyone the company employs. Your QA process tests the ten or twenty percent you wrote. It has essentially nothing to say about the rest, because "nothing to say" is by design — a unit test doesn't know or care whether the left-pad function it's calling was published by the actual maintainer of left-pad or by an attacker who compromised that maintainer's npm account six hours ago. The test only knows whether the function pads a string on the left. If the attacker's version does that correctly and also exfiltrates environment variables, your test suite passes and your static analyzer, which is scanning your code, never looks at the dependency at all.

This is the structural reason provenance verification keeps falling through the cracks between security, platform engineering, and QA. Security teams tend to own vulnerability management — scanning for known CVEs in dependencies — which is a related but distinct problem from provenance. A dependency can have zero known vulnerabilities and still not be the artifact its maintainer actually published. Platform teams tend to own the CI/CD pipeline's mechanics — build speed, caching, deployment orchestration — without necessarily owning a mandate to verify what's flowing through it. QA teams tend to own correctness testing, which by definition assumes the thing under test is legitimate to begin with. Provenance verification sits in the gap between all three, which is exactly why it's frequently owned by no one.

What Provenance Verification Actually Checks — and What It Doesn't

It's worth being precise about terms here, because "supply chain security" has become a broad enough phrase to mean almost anything, and vague framing is part of why this work doesn't get prioritized.

Dependency provenance verification is not the same as vulnerability scanning. Software composition analysis (SCA) tools like Dependabot, Snyk, or Sonatype's scanning products check whether the packages in your dependency tree have publicly disclosed vulnerabilities (CVEs) and, increasingly, whether known-malicious packages have been identified in threat intelligence feeds. That is valuable and most organizations already do at least some of it. But SCA answers "is this a known-bad package?" Provenance verification answers a prior and different question: "is this package what it claims to be, and can I prove that without depending on it having already been caught and cataloged by someone else?"

Provenance verification, as a testable release property, breaks down into four distinct, checkable claims:

Identity. The package came from the source it claims to come from — a specific maintainer, a specific repository, a specific organization — and that identity is cryptographically verifiable rather than asserted by a username string that anyone could type into a registry's metadata field.

Integrity. The bytes you are about to compile or install are exactly the bytes the maintainer published, with no alteration introduced between publication and your build — whether by a compromised registry, a man-in-the-middle mirror, a poisoned cache, or a tampered lockfile.

Build fidelity. If the artifact was built from source rather than distributed as source, the build process that produced the artifact is documented, reproducible in principle, and didn't run on infrastructure an attacker could have quietly modified.

Change legitimacy. An update to a dependency — a new version, a new maintainer added to a project, a new transitive dependency introduced — went through some form of scrutiny proportional to its risk, rather than being pulled in automatically the moment it appears on a registry.

None of these four claims is addressed by code review of your own repository, by unit tests, or by static analysis of your own source. All four are addressed by the specific practices this article covers: software bills of materials, build provenance attestation, lockfile integrity enforcement, and risk-based dependency review policy. Each is testable. Each can fail a build. That's the argument in one paragraph — the rest of this article is about how to actually build it.

Five Incidents That Show the Gap Is Not Theoretical

Skeptics of this argument sometimes suggest that provenance attacks are rare, sophisticated, nation-state-level events that don't justify pipeline changes for an ordinary product team. The public incident record doesn't support that. Below are five real, independently documented cases spanning different registries, different attack techniques, and different levels of sophistication — from a years-long infiltration campaign to a compromised bot token. None of these are hypothetical; each is drawn from CISA advisories, vendor incident write-ups, or the affected registry's own postmortem.

The xz-utils backdoor (2024)

xz-utils is a compression library used across essentially every mainstream Linux distribution, embedded deep enough in the dependency graph that most engineers who rely on it have never heard of it. In early 2024, a contributor using the name "Jia Tan" — who had spent roughly two years building trust with the project's sole maintainer, submitting legitimate patches and gradually taking on co-maintainer responsibilities — introduced obfuscated malicious code into versions 5.6.0 and 5.6.1 of the library. The backdoor was designed to intercept and manipulate SSH authentication (via a dependency chain running through liblzma into sshd on affected distributions), potentially granting an attacker remote code execution on any system running the compromised versions.

The backdoor was discovered by Andres Freund, a PostgreSQL developer, who noticed that SSH logins on a system he was benchmarking were consuming noticeably more CPU time than expected — roughly half a second of unexplained latency. That observation, and Freund's decision to investigate an anomaly most engineers would have shrugged off, is very plausibly the reason this did not become one of the most damaging supply chain compromises in the history of open source software. CISA issued a formal advisory (CVE-2024-3094), confirmed the affected versions, and recommended downgrading to xz-utils 5.4.6 — a known-uncompromised release — while investigating for signs of exploitation.

What makes this case instructive for a provenance-testing argument is not the sophistication of the payload. It's that the compromise succeeded entirely through the normal, expected mechanics of open-source maintainership — a real contributor, a real commit history, a real release process — with no exploit of any technical vulnerability required to get the malicious code into the official release artifact. No amount of code review of your own application would have caught this, because the malicious code never touched your repository. It arrived through your build's dependency resolution, indistinguishable at that layer from any other point release.

The event-stream / flatmap-stream incident (2018)

event-stream was a popular npm utility library with millions of weekly downloads. Its original maintainer, no longer actively developing it, accepted an offer from a new contributor to take over maintenance — a common and usually benign pattern in open source. The new maintainer subsequently published a version of event-stream that added a new dependency, flatmap-stream, which itself contained encrypted malicious code. According to npm's own incident write-up, the payload was specifically targeted: it activated only inside the build process of the Copay Bitcoin wallet application, where it attempted to locate and exfiltrate private keys and account credentials. Because the targeting was narrow, the malicious package sat in the open registry, downloaded by unrelated projects with no ill effect on them, for roughly two and a half months and an estimated eight million downloads before Copay's own team identified the anomaly and reported it. npm removed the package and published a detailed postmortem.

This incident established a pattern that has recurred many times since: maintainer handoff as an attack vector. A project with real trust and real usage changes hands, quietly, through a mechanism (a friendly email offering to help) with essentially no external verification. Nothing about that transaction is visible to a downstream consumer's build pipeline unless something is specifically watching for it.

The ua-parser-js account takeover (2021)

ua-parser-js, a widely used npm library for parsing browser user-agent strings, was compromised in October 2021 when an attacker gained control of the maintainer's npm account — not through a vulnerability in the package itself, but through account credential compromise. The attacker published several new versions containing a cryptocurrency miner and, on Windows systems, a password-stealing trojan. Because ua-parser-js was a transitive dependency of an enormous number of projects (Rapid7's incident write-up noted downstream usage by major technology companies, illustrating how deep and wide a single popular package's reach can be), the blast radius of a short window of compromise was large. The maintainer regained control of the account, npm deprecated the malicious versions, and the incident became a widely cited case study in why maintainer account security is a supply-chain control, not just an individual developer's personal responsibility.

The tj-actions/changed-files GitHub Actions compromise (2025)

This incident moves the threat model from package registries to CI/CD pipelines directly, which is directly relevant to the pipeline-hardening argument later in this article. tj-actions/changed-files is a popular third-party GitHub Action used inside CI workflows to detect which files changed in a pull request — the kind of small, convenient utility Action that gets referenced by tag (@v45, for example) in thousands of repositories' workflow files without much scrutiny. In March 2025, an attacker compromised a GitHub personal access token belonging to a bot account with write access to the Action's repository, and used it to retroactively rewrite the Action's version tags to point at a malicious commit — meaning that existing workflows referencing what they believed was a stable, previously reviewed version suddenly started executing different code the next time they ran, with no change made on the consuming side at all.

The malicious code dumped CI runner memory contents, looking for secrets, and printed them — double-base64-encoded — into the public build logs of any repository that ran the action, which in public repositories meant the secrets became visible to anyone who could view the log output. CISA issued an advisory (CVE-2025-30066, alongside a related compromise of reviewdog/action-setup, CVE-2025-30154). Security researchers at StepSecurity identified the compromise, and the maintainer reverted the malicious commits and removed the compromised tags within roughly a day of initial reports. Wiz's incident analysis noted that leaked credentials in some affected repositories included AWS access keys, npm publishing tokens, GitHub PATs, and private RSA keys.

This case demonstrates a distinct failure mode from a compromised package: it's a compromised reference. Nothing in the consuming repository's own dependency file changed. The vulnerability lived entirely in the fact that a version tag — which most engineers treat as an immutable pointer, the way they'd treat a Git commit hash — is not actually immutable on GitHub, and can be silently rewritten by anyone with sufficient access to the upstream repository.

Dependency confusion research (2021)

Unlike the previous four cases, this one was disclosed responsibly by a security researcher rather than exploited maliciously, but it exposed a structural flaw affecting an enormous number of organizations simultaneously. Security researcher Alex Birsan discovered that many companies used internal, privately named packages (for build tooling, internal libraries, and so on) without ever publishing a same-named package to the public registry to reserve the name. Package managers in several ecosystems, when resolving a dependency, would in some configurations prefer whichever version had the higher semantic version number — including a public package with the same name as an internal one, if the public version number was higher. Birsan published proof-of-concept packages under internal package names he'd discovered (through build logs, GitHub repos, and other public leakage of internal naming conventions) for more than 35 companies, including Apple, Microsoft, PayPal, Netflix, Tesla, and Uber, and documented that his benign test code executed inside those companies' internal build systems, earning him bug bounty payouts exceeding $130,000 combined. No malicious payload was ever involved — Birsan's packages simply "phoned home" to confirm execution — but the technique, once published, was rapidly weaponized by other actors against less careful targets.

This incident matters for a different reason than the others: it shows that a provenance failure doesn't require any compromise of a real maintainer's account or infrastructure at all. It can arise purely from a naming ambiguity between your organization's internal namespace and the public registry namespace, resolved silently and automatically by tooling behaving exactly as designed.

What these five cases have in common is that none of them would have been caught by code review of the consuming organization's own repository, by that organization's unit tests, or by static analysis of code that organization wrote. Each required a form of verification aimed specifically at the dependency layer — cryptographic identity checking, build provenance attestation, pinned and hash-verified references, or namespace reservation — that most pipelines, even mature ones, simply never implement.

A Maturity Model for Dependency Provenance

Because this practice area sits between security, platform engineering, and QA ownership, organizations tend to have wildly inconsistent maturity even within a single company — one team might sign their releases while another, in the same organization, resolves dependencies from a public registry with no lockfile at all. The following model gives engineering leaders a way to place their pipeline on a five-level scale and identify the next concrete step, rather than treating "supply chain security" as a binary you either have or don't.

A few things about this model are worth stating explicitly, because maturity models like this one are easy to misuse. First, most organizations reading this are somewhere between Level 0 and Level 1 — a 2024 Sonatype survey-based report found that a majority of organizations still lack a documented policy for evaluating open-source component risk before adoption, and lockfile enforcement in CI, while common, is frequently undermined by build scripts that regenerate the lockfile rather than failing when it's out of sync. Second, moving up a level is not free, and the cost curve is not linear — Level 2 to Level 3 in particular requires tooling changes across every language ecosystem in use, not a single tool adoption. Third, and most important: this is not a compliance checklist to complete once. Each level's controls need to run on every release, which is why the framing throughout this article is "testable release property" rather than "security initiative."

Generating and Verifying a Real SBOM

A software bill of materials is, at minimum, a structured, machine-readable inventory of every component — direct and transitive — in a given build, along with enough metadata (version, source, license, sometimes cryptographic hash) to identify each one unambiguously. The concept predates its current prominence by decades (manufacturing has used bills of materials for physical products for a very long time), but it became a formal software engineering requirement following the U.S. Executive Order 14028 on Improving the Nation's Cybersecurity (2021), which directed NIST to develop minimum elements for SBOMs and directed federal agencies toward SBOM-related procurement requirements for software vendors, formalized through OMB guidance (M-22-18) and NIST's Secure Software Development Framework (SSDF).

It's worth noting, in the interest of giving engineering leaders current and accurate regulatory context rather than a stale compliance narrative: OMB guidance in this area is not static. A February 2026 OMB memorandum (M-26-05) rescinded the prior government-wide mandate requiring agencies to collect standardized self-attestations and gave individual federal agencies discretion over whether and how to require SBOMs from vendors, replacing a centralized requirement with an agency-by-agency, risk-based approach. The practical implication for a vendor selling into the federal market is that SBOM requirements have become less uniform, not that they've disappeared — some agencies will still require them, using CISA's common attestation form or their own variant, and any vendor serious about that market segment should expect an SBOM request to keep surfacing on due-diligence and procurement calls regardless of the exact regulatory mechanism in force at a given moment. Outside government procurement, enterprise customers running their own vendor risk assessments increasingly ask for an SBOM regardless of what federal policy currently requires, simply because it's now normal due-diligence practice — which is closer to the actual driver behind most companies' SBOM adoption than compliance with any specific regulation.

Two open specifications dominate SBOM generation in practice, and the choice between them is not cosmetic.

Neither format is objectively superior; they were built to solve related but different original problems, and a growing number of organizations generate both, since several toolchains (notably Anchore's syft) can produce either format from the same scan with a flag change. The practical decision for most engineering teams is: if your primary driver is vulnerability management and your security tooling already speaks CycloneDX (most SCA vendors do), start there; if your primary driver is regulatory submission requiring ISO-standard documentation or detailed license provenance, SPDX has the edge. Either choice is far better than the common default of no SBOM at all.

Generating an SBOM is the easy part — most build tools have a plugin or CLI flag for it now. The part that actually makes it a testable release property is what you do with it afterward, and this is where most organizations stop short:

1. Generate the SBOM as a build artifact, not a side report. It should be produced automatically as part of the same pipeline run that produces the deployable artifact, stored alongside it, and versioned identically — an SBOM that was generated manually, after the fact, for a compliance questionnaire tells you nothing reliable about what actually shipped.
2. Diff the SBOM against the previous release's SBOM as an automated CI step. A new transitive dependency appearing between two releases, especially one introduced by an update to something several levels removed from your direct dependencies, is exactly the kind of change that deserves a human glance and currently gets none in most organizations, because nobody is looking at the dependency tree at all outside of the occasional npm audit run.
3. Fail the build, don't just log a warning, when the SBOM can't be generated. If SBOM generation silently fails (a common failure mode when a private registry or an unusual build step confuses the generator), a "documented" pipeline quietly becomes an "unmanaged" one for that release, with nobody noticing until someone asks for the SBOM and it doesn't exist.
4. Feed the SBOM into vulnerability scanning as the source of truth, rather than running a separate ad hoc dependency scan that might resolve a slightly different tree than what actually got built and deployed.
5. Retain SBOMs for the full support lifetime of the release, not just the current version — when a new CVE or supply chain incident is disclosed for a package you used eighteen months ago, an archived SBOM lets you answer "were we affected, and in which releases" in minutes rather than days of git archaeology.

A Worked Example: What an SBOM Diff Actually Catches

Because "diff the SBOM between releases" can sound abstract until you've seen what the output looks like, it's worth walking through a concrete, illustrative example. The following scenario is hypothetical — a constructed teaching example, not a real company or a real incident — but the mechanics are exactly how this control behaves in practice.

Imagine a mid-sized fintech application's payment-reconciliation service, releasing weekly. The previous release's SBOM lists 214 total components (18 direct dependencies, 196 transitive). The current release candidate's SBOM lists 217 components. On the surface, that's an unremarkable three-package increase — the kind of routine drift that happens on almost every release and that nobody manually inspects in an organization without an enforced diff step.

A diff step run against these two SBOMs would surface exactly three lines:

In an organization without this diff step, all three of these changes ship silently as part of what looks, from the pull request description alone, like a routine "bump dependency X to latest patch version" change — exactly the kind of update that a risk-tiering policy without SBOM visibility would auto-accept, because the direct dependency's version change is a minor patch bump, which is precisely the case most auto-accept policies are designed to wave through without friction.

With the diff step in place, the second line — a newly introduced transitive dependency carrying an install-time script, from a package with a short publication history — is exactly the kind of signal the review checklist earlier in this article is built to catch. It doesn't prove anything malicious is happening. Most of the time, a new install script on a young package is completely benign — many legitimate packages compile native bindings or set up local configuration at install time. The value of the diff step is not that it produces a verdict; it's that it produces visibility into a change that would otherwise be invisible, and hands a specific, bounded question to a human reviewer — "does this specific new script do what it claims to do" — instead of leaving that judgment to be exercised, if ever, only after something has already gone wrong.

This is also the clearest illustration of why SBOM generation without diffing is a materially weaker control than SBOM generation with diffing. An SBOM that exists only to be handed to an auditor or a due-diligence reviewer on request answers "what's in this build" after the fact. An SBOM that's diffed automatically on every release, with new install scripts and new maintainers flagged as part of that diff, answers the more useful question — "what changed, and does the change deserve a second look" — at the moment a human can still act on it before the artifact reaches production.

Build Provenance Attestations: SLSA and Sigstore in Practice

An SBOM tells you what's in a build. It does not, by itself, prove that the build itself wasn't tampered with, or that the artifact you're about to deploy was actually produced by the build process it claims to have come from. That's a different claim, and it requires a different mechanism: cryptographically signed build provenance.

SLSA (Supply-chain Levels for Software Artifacts, pronounced "salsa") is an open framework, originally developed at Google and now maintained as an Open Source Security Foundation project, that defines a graduated set of requirements for how much you can trust that a given artifact was built the way its provenance metadata claims. The framework's official specification (version 1.0) defines four build levels:

Reaching SLSA L3 is a meaningful infrastructure commitment — it typically means using a build platform specifically designed to provide these isolation guarantees (several major CI providers now offer SLSA L3-capable generators for common ecosystems) rather than a general-purpose CI runner configured by hand. For most organizations, L1 or L2 represents a realistic near-term target that already closes off a meaningful set of attack paths, and is worth pursuing before chasing L3 as an abstract goal.

Sigstore is the open-source project that has become the practical mechanism for actually implementing signed provenance without the traditional operational burden of long-lived signing keys. Sigstore's core insight, documented in its official project materials, is that most organizations historically avoided artifact signing not because they didn't understand its value, but because managing private signing keys securely — rotating them, protecting them from exfiltration, revoking them when a key holder leaves — is its own significant operational burden. Sigstore's cosign tool implements "keyless signing": instead of a long-lived private key, a signer authenticates via an OpenID Connect identity provider (GitHub, Google, or a corporate SSO provider), and Sigstore's certificate authority component, Fulcio, issues a short-lived certificate binding that verified identity to a temporary signing key generated on the spot and discarded immediately after use. Every signing event is recorded in Rekor, an append-only, publicly auditable transparency log, so that even if someone later disputes a signature, there's an immutable public record of when it was created and under which identity.

The practical effect for a release pipeline: your CI system, running as an authenticated GitHub Actions workflow (for example), can sign a build artifact using its workflow identity, with no signing key ever stored in a secrets manager, no key rotation schedule to maintain, and a public, timestamped record that the signature happened. Verification, downstream, checks the signature against Rekor and confirms the identity that produced it — which is a meaningfully different guarantee from "an artifact with this checksum was uploaded to this registry," because it ties the artifact to a specific, auditable build identity rather than just to a hash that a compromised process could recompute for a tampered file just as easily as for a legitimate one.

Both npm and PyPI have adopted Sigstore-based mechanisms directly into their publishing flows, which matters because it lowers the barrier for individual package maintainers, not just large platform teams, to participate:

• npm provenance attestations (available via npm publish --provenance when publishing from a supported CI environment such as GitHub Actions) automatically generate and attach a signed SLSA-style provenance statement to a published package, linking it to the exact source repository, commit, and workflow run that produced it. npm displays a verified "Provenance" badge on the package page when this is present, and downstream consumers can verify it with the npm CLI or independently.
• PyPI Trusted Publishers, introduced via an official PyPI feature launched in 2023, lets a maintainer configure a specific CI workflow (again, commonly GitHub Actions) as an authorized publisher for a package using OIDC, eliminating the need for a long-lived PyPI API token stored as a CI secret at all — which directly removes one of the most common ways a publishing credential gets leaked or stolen in the first place, since there's no static token to leak.

A simplified view of what a signed build-and-verify flow looks like in practice:

The gate that actually matters is the diamond in that diagram — verification on the consuming side. An organization can adopt every tool in this section and still gain nothing if the CI pipeline pulling in dependencies never actually checks the signatures it receives; a huge number of real-world "we use Sigstore" claims turn out, on inspection, to mean "we generate signatures" without a corresponding step anywhere that fails the build when a signature is missing or invalid. Generation without enforcement is theater. The release gate is the enforcement step, not the signature's mere existence.

Lockfile Integrity as a Release Gate

Long before SBOMs and signed attestations became mainstream conversation topics, most package ecosystems already had a mechanism that, if actually enforced, closes off a meaningful share of the risk described in this article: the lockfile. package-lock.json and yarn.lock for npm/Yarn, Pipfile.lock or poetry.lock for Python, Gemfile.lock for Ruby, go.sum for Go — all of these record not just a resolved version number for every dependency, but, in most implementations, a cryptographic hash of the exact package contents that were resolved at lock time.

The gate this enables is straightforward in principle: a build step that installs strictly from the lockfile, verifying every hash, and fails rather than silently falling back to a fresh resolution when something doesn't match. In practice, this control fails in three common, entirely mundane ways that have nothing to do with sophisticated attacks:

The build uses the wrong install command. npm install will happily update the lockfile to satisfy a version range if the exact locked version isn't available or if package.json has changed; npm ci will not — it installs exactly what's locked and fails if the lockfile and package.json are out of sync. Many CI configurations, often inherited from a local development script rather than deliberately chosen for CI, use the former. The distinction sounds trivial and is nonetheless one of the most common gaps found during a pipeline maturity assessment.

The lockfile isn't actually committed, or is regenerated as part of the build. Some build scripts, particularly older ones or ones ported from a simpler era of a project's life, run a fresh dependency resolution as a build step "to make sure everything's up to date" — which defeats the entire purpose of a lockfile by re-resolving against whatever happens to be on the registry at build time rather than the reviewed, previously-resolved set of versions.

Hash verification is available but not enabled. Python's pip, for instance, supports --require-hashes, which forces every package installation to match a hash listed in the requirements file — but this flag is opt-in, and a large share of Python projects don't use it even when a lockfile-equivalent mechanism (via pip-compile or Poetry) is otherwise in place.

None of these three gaps require sophisticated tooling to close. They require a five-minute pipeline configuration review and, in most cases, a one-line change to a CI script. The fact that they persist in a large number of production pipelines despite costing almost nothing to fix says something important about the underlying problem this article is describing: it's not that provenance verification is expensive or hard, in every case. It's that nobody's job description includes checking for it, so a control that already exists in the tooling sits unused, unmentioned in any runbook, until an incident or an audit forces someone to look.

Lockfile integrity is not a substitute for SBOMs or signed attestations — a hash-verified install still trusts that the hash in the lockfile itself was legitimate when it was committed, which is a weaker guarantee than an independently signed, transparency-logged attestation. But it is the cheapest, fastest control in this entire article to implement, it catches a meaningful share of tampering and registry-substitution attacks, and there is close to no excuse for an organization at Level 1 in the maturity model above to remain at Level 0.

A Risk-Based Review Policy for Dependency Updates

Automated verification — SBOMs, signatures, lockfile hashes — catches tampering and forgery. It does not, by itself, catch a legitimate maintainer making a decision you would have objected to if you'd known about it in advance: adding a new, unreviewed transitive dependency; introducing a post-install script that executes arbitrary code on every consumer's machine; or, as in the event-stream case, handing maintainership to someone new. These are legitimate, correctly-signed, hash-verified changes that a purely mechanical verification pipeline will wave through every time, because nothing about them is technically fraudulent.

Closing this gap requires a review policy — not a policy that scrutinizes every dependency update equally (which is neither realistic nor a good use of scarce reviewer attention), but one that tiers scrutiny by risk. A workable framework asks a small number of concrete questions about any proposed update, before merge:

Does this update introduce a new maintainer or transfer of ownership? Registries generally expose this metadata; a script that flags a maintainer change on any dependency, direct or transitive, for manual review before the update is accepted is one of the highest-leverage, lowest-effort controls available, given that maintainer handoff is a documented factor in multiple real incidents, including the event-stream case above.

Does the package (or an updated version of it) run install-time scripts? postinstall hooks in npm, and their equivalents in other ecosystems, execute arbitrary code on the machine performing the install — which, in a CI environment, is your build infrastructure, with whatever access that infrastructure has. A package that previously had no install script and gains one in an update is a meaningfully different risk profile than a routine version bump, and deserves a different level of scrutiny.

How deep in the tree is the change, and how widely does the affected package propagate? A version bump to a leaf dependency used by exactly one internal tool carries different risk than an update to a package sitting at the root of dozens of other dependencies' trees — the tj-actions incident is instructive here precisely because the compromised component was small, low-profile, and used almost everywhere, which is a common profile for high-impact supply chain targets rather than an unusual one.

Is the update inside or outside the range your policy allows without review? A patch version bump on a mature, widely used package is a reasonable candidate for automatic acceptance once signature and lockfile checks pass. A major version bump, a jump across multiple versions at once, or an update to a package with a short history and low adoption is not, and treating them identically is a common and avoidable failure of most automated dependency-update tools' default configurations.

Does the update change license terms? Not a security question in the narrow sense, but a real release-blocking risk in commercial software, and one that a good SBOM diff process (described earlier) surfaces automatically if anyone is actually looking at the diff.

A practical checklist, usable as a merge-gate template for a dependency-update pull request:

• Automated signature/provenance verification passed (not just "ran" — actually returned a pass)
• Lockfile hash verification passed for every changed and newly introduced package
• SBOM diff reviewed; any new transitive dependency is accounted for
• No new maintainer or ownership change on any updated package, or one is present and has been specifically reviewed
• No new install-time script introduced, or one is present and its contents have been read by a human
• Version jump is within the policy's auto-accept range, or an explicit reviewer sign-off is attached for anything larger
• License of the updated package(s) has not changed in a way that affects distribution terms
• Update is applied to a non-production environment first and observed before promotion, consistent with normal release practice for any change

This checklist is deliberately short. A review policy that takes twenty minutes to apply to every routine dependency update will be abandoned within a quarter; one that takes two minutes for the ninety percent of updates that are routine, and escalates the remaining ten percent for real scrutiny, is the version that survives contact with an actual engineering team's workload.

Hardening the CI/CD Pipeline Itself

Everything described so far assumes the pipeline doing the verifying is itself trustworthy. The tj-actions incident is a useful reminder that the pipeline is part of the supply chain too, not just an observer of it — and pipeline-layer compromises have a specific set of countermeasures distinct from dependency-layer ones.

Pin third-party CI actions and reusable workflows by commit SHA, not by tag or branch name. A Git tag on most platforms, including GitHub, is mutable by default — anyone with sufficient write access to the upstream repository can move it to point at a different commit, exactly as happened in the tj-actions compromise. A commit SHA is not mutable in the same way; pinning to one means a compromised tag can't silently change what your workflow executes. This is a genuinely low-cost change (most CI providers support it natively) with a direct, documented incident behind the recommendation.

Enforce least-privilege tokens for every CI job, scoped to exactly what that job needs. GitHub Actions, for instance, defaults workflow tokens to broad read/write permissions unless explicitly restricted; a job that only needs to read repository contents and post a comment on a pull request should not hold a token capable of pushing tags, publishing packages, or modifying repository settings. The tj-actions incident's damage was amplified by the range of secrets accessible to the compromised bot token; a narrower scope limits blast radius even when a compromise happens.

Maintain an explicit allowlist of third-party Actions and equivalent CI marketplace components, rather than allowing any workflow author to reference any published Action. This doesn't need to be bureaucratic — a short, centrally maintained list of pre-approved, pinned actions covers the overwhelming majority of legitimate use cases in most organizations, and anything outside the list goes through a brief review before adoption, which is the same risk-tiering principle applied to pipeline tooling rather than application dependencies.

Separate build and deploy credentials, and avoid granting deploy credentials to any job that executes third-party code as part of its normal function. A build step that runs a linter, a test suite that includes third-party test fixtures, or an Action that processes pull request content from external contributors is, definitionally, executing code you don't fully control; it should never hold the credentials needed to push to production.

Pursue reproducible builds where the toolchain realistically supports them. A reproducible build is one where, given the same source and the same build inputs, independent parties can produce bit-for-bit identical output — which means a build's provenance attestation can be independently verified by re-running the build, rather than trusted solely on the strength of a signature from the original build infrastructure. Full reproducibility is a genuinely hard engineering property to achieve for most real-world toolchains (compilers that embed timestamps, non-deterministic dependency resolution order, and file-system metadata are common obstacles), and it is not a reasonable near-term target for every organization. Where it is achievable — several language ecosystems and the Reproducible Builds project (a long-running, cross-distribution effort originating in the Debian and Tor communities) have made real, documented progress here — it provides a form of verification that doesn't depend on trusting any single party's infrastructure, including your own.

Ecosystem-by-Ecosystem: Where the Playbook Changes

Everything described so far applies broadly across ecosystems, but the specific mechanics — and the specific gaps — differ enough between package managers that a single, undifferentiated rollout plan tends to underserve at least one language in a typical polyglot organization's stack.

The practical implication for a platform team rolling this out across a polyglot organization: sequence the rollout by ecosystem-specific leverage, not uniformly. An organization with a large Go codebase already inherits a strong default posture from the toolchain itself and should prioritize its npm and Python services, where opt-in controls are more commonly left disabled. An organization running containerized workloads should treat digest pinning for base images with the same urgency as commit-SHA pinning for CI actions, since it's the same underlying vulnerability — a mutable reference standing in for an immutable one — expressed in a different layer of the stack.

What to Measure, and What Not to Mistake for Progress

Every practice described in this article can be measured, and every measurement can also be gamed or misread if adopted without thought. This distinction matters because engineering leaders reporting on this work to a board, an acquirer, or their own leadership need metrics that reflect actual risk reduction rather than activity.

Useful metrics tend to measure coverage and enforcement, not just existence:

• Percentage of production releases with a generated, archived SBOM (not "we have SBOM tooling installed" — the actual per-release coverage rate)
• Percentage of production releases with a verified (not merely generated) build provenance signature checked before deployment
• Percentage of CI jobs using least-privilege, job-scoped tokens versus broad default tokens
• Mean time between a disclosed dependency-related incident (a CVE, a registry advisory, a maintainer compromise report) affecting a package in your tree and your organization's confirmed determination of exposure — this measures whether your SBOM archive is actually usable under time pressure, not just present
• Number of dependency updates that triggered manual review under your risk-tiering policy versus number auto-accepted, tracked over time to confirm the policy is neither so loose it never fires nor so strict it gets bypassed

Misleading metrics, common in vendor dashboards and easy to over-index on:

• "Number of vulnerabilities scanned" or "number of dependencies scanned" as a standalone figure — this measures scan volume, not risk reduction, and a scan that runs against an outdated or incomplete dependency tree produces a large, reassuring-looking number that reflects very little
• "SBOM generated: yes/no" as a binary organizational claim, when the actual per-release generation rate might be well under 100 percent, or the SBOM might not be verified for completeness against what was actually deployed
• Raw count of CVEs "fixed" per quarter, which rewards teams for having had more vulnerabilities to begin with and says nothing about whether the fixes addressed exploitable, reachable code paths versus unreachable ones
• Adoption of a specific named tool or vendor product as a proxy for maturity — tool adoption is necessary but not sufficient; the maturity model in this article is defined by verification and enforcement behavior, not by which logo appears in the CI configuration

Where This Practice Breaks Down, and for Whom

No control described in this article is free, and none of it eliminates risk entirely — a useful maturity model has to be honest about its own limits, or it becomes another source of false confidence rather than a corrective to one.

For an early-stage startup with three engineers and no dedicated platform function, pursuing SLSA L3 and a formal risk-tiering committee for dependency updates is not a sensible use of scarce time. Lockfile enforcement (npm ci instead of npm install in CI) and an SBOM generated on every release, even without signature verification, are achievable in an afternoon and close off a meaningful share of the risk described here at near-zero ongoing cost. The full maturity model is a map for where to grow into, not a bar every organization needs to clear immediately.

For a scale-up preparing for enterprise sales or acquisition due diligence, the calculus changes — an SBOM you can produce on request, and a documented (even if informal) update review practice, increasingly show up as explicit questions in security questionnaires and due-diligence checklists, independent of whether any incident has ever occurred. This is where the gap described in this article's opening scenario tends to surface first, often at the worst possible time.

For a regulated enterprise (financial services, healthcare, critical infrastructure), signed build provenance and a governed, auditable review policy are increasingly close to table stakes, driven as much by customer and counterparty risk assessments as by any specific regulatory text, and the cost of building this capability under audit pressure, after a finding, is reliably higher than building it deliberately in advance.

There's a fourth category worth naming separately, because it's the one this article opened with: an organization mid-acquisition or mid-enterprise-sale, where the question isn't really "how much risk have we reduced" but "can we produce evidence of the practice on demand, under time pressure, to a counterparty who doesn't take our word for it." This is a subtly different requirement from the other three. A startup optimizing for genuine risk reduction on a tight budget might reasonably defer signed attestation in favor of cheaper controls. An organization facing a due-diligence timeline cannot defer the same way, because the ask isn't "reduce your risk" — it's "prove your current state," on a calendar that due diligence sets, not one the engineering team controls. Building SBOM generation and a documented review policy before that conversation starts, rather than during it, is the difference between a due-diligence call that takes twenty minutes and one that takes three weeks of retroactive archaeology through git history and closed pull requests.

It's also worth stating plainly where this discipline does not help. It does not protect against a vulnerability in code your own team wrote — that remains the job of your existing QA and security practices. It does not protect against a zero-day disclosed in a dependency the same day you deploy it, before any advisory exists to detect against — provenance verification confirms authenticity and origin, not the absence of undiscovered bugs. And it does not eliminate the judgment calls inherent in software engineering — a correctly signed, hash-verified, provenance-attested package can still contain a design decision your team would have made differently, and no amount of cryptographic verification substitutes for an engineer actually reading a diff when the stakes warrant it.

An Implementation Sequence

For a team starting from Level 0 or Level 1 in the maturity model above, the following sequence reflects both dependency ordering (some steps require others as prerequisites) and cost-to-value ratio (cheaper, higher-impact steps first):

1. Audit current lockfile enforcement across every service and repository. Confirm CI actually installs from the lockfile with hash verification where the ecosystem supports it, rather than assuming it does because a lockfile file exists in the repository.
2. Generate an SBOM on every release build, in CycloneDX or SPDX depending on whether your primary driver is security tooling integration or compliance documentation, and store it as a versioned release artifact alongside the deployable.
3. Add an automated SBOM diff step that flags new or changed transitive dependencies between releases for a lightweight human check, rather than silent acceptance.
4. Adopt commit-SHA pinning for all third-party CI actions and reusable workflows, replacing tag or branch references across existing pipelines — this is mechanical, scriptable work and does not require new tooling.
5. Audit and reduce CI token scopes to least privilege, job by job, prioritizing any job with deploy, publish, or secret-access capability.
6. Enable build provenance signing where your ecosystem's publishing flow supports it natively (npm's --provenance flag, PyPI Trusted Publishers) before attempting a more general Sigstore/cosign integration for internally built artifacts.
7. Add signature verification as an enforced, build-failing gate on the consuming side — this is the step most commonly skipped, and the one that actually converts signing from a nice-to-have into a real control.
8. Draft and socialize a risk-tiered dependency update review policy, using the checklist in this article as a starting template, and calibrate the auto-accept range based on your actual update volume and reviewer capacity.
9. Establish an incident response runbook specific to dependency compromise, distinct from your general security incident runbook, covering how you'd use your SBOM archive to determine exposure if a package you depend on is disclosed as compromised.
10. Report provenance-specific metrics (from the metrics section above) alongside your existing release-health dashard, on a recurring cadence, so this becomes a maintained practice rather than a project that concludes.

Questions for Engineering Leaders to Ask Their Own Teams

A due-diligence evaluator, a board member, or a CTO auditing their own organization can use a short set of direct questions to establish, in a single conversation, roughly where a pipeline sits on the maturity model:

• "Can you produce the SBOM for the exact build currently running in production, right now, without generating a new one first?"
• "If we told you today that a package you depend on was just disclosed as compromised, how long would it take you to determine whether you're affected, and in which releases?"
• "Do any of our CI workflows reference third-party Actions by tag rather than commit hash, and if so, how many?"
• "What CI token scopes does our deploy pipeline actually hold, and has anyone reviewed whether it needs all of them?"
• "When was the last time a dependency update was rejected or flagged for review by our process, and why?"
• "If a maintainer of one of our critical dependencies changed today, would we find out, and from what signal?"

An engineering leader who can answer all six of these with specifics, rather than "I'd need to check," is operating well above the industry median. An organization where none of these have ever been asked internally is, per the incidents documented earlier in this article, not hypothetically exposed — it is exposed in exactly the way real, publicly disclosed organizations already have been.

The Discipline This Requires

Dependency provenance testing is not a security team's side project, and it is not a compliance checkbox satisfied once and revisited at the next audit cycle. It is a release-engineering discipline with the same shape as any other quality gate your organization already runs: a specific, falsifiable claim ("this artifact is what it claims to be, from who it claims to be from, unaltered since publication"), a mechanical way to check that claim, and a decision to block release when the check fails. The tooling to do this — CycloneDX and SPDX for inventory, SLSA and Sigstore for attestation, lockfile hash verification for integrity, risk-tiered review for judgment calls that automation can't make — is open, mature, and in active use by major package registries and build platforms today. What's missing in most organizations isn't the technology. It's the decision to treat this as a release property with an owner, a gate, and a measured pass rate, the same way test coverage or deployment success rate already are.

The next time your organization is asked, in a due-diligence call, a security questionnaire, or an internal postmortem after someone else's incident makes the news, whether you can prove what's actually running in production and where it came from — that question will already have an answer, or it won't. Building the answer in advance is considerably cheaper than building it under pressure.

If there's one distinction worth carrying back to your own team from this article, it's the one drawn in the opening section: testing your own code and verifying your dependencies' provenance are not the same discipline, and treating them as though the first one covers the second is the gap every incident in this article walked straight through.

How QAtronic can help: Organizations bringing QAtronic in to strengthen release and QA practices increasingly ask us to assess pipeline maturity against exactly this kind of framework — not to run a one-time audit, but to help build SBOM generation, provenance verification, and dependency review policy into the same release gates that already govern test coverage and deployment approval. If your team can't currently answer the due-diligence question this article opened with, that gap is a reasonable, well-scoped starting point for a conversation.

Frequently Asked Questions

Is a software bill of materials the same thing as a vulnerability scan? No. An SBOM is an inventory — it lists what's in a build. A vulnerability scan checks that inventory (or a separately resolved dependency tree) against known-CVE databases. You need both: the SBOM makes the scan trustworthy by ensuring it's checking what actually shipped, and the scan makes the SBOM actionable by telling you which listed components carry known risk.

Does adopting SLSA or Sigstore require replacing our existing CI/CD platform? Generally no. Sigstore's tooling (cosign) integrates with most major CI platforms without a platform migration, and both npm and PyPI's native provenance features work with standard GitHub Actions workflows specifically. Reaching SLSA's higher build levels (particularly L3) can require a build platform designed for stronger isolation guarantees, but L1 and L2 are achievable on most existing CI infrastructure with configuration changes rather than a platform switch.

We already use Dependabot or Renovate for automated dependency updates. Doesn't that cover this? Partially, and it's a good foundation, but automated update tools primarily solve staleness and known-vulnerability remediation — they open a pull request when a newer or a patched version exists. They generally do not verify build provenance signatures, generate SBOMs, or apply risk-tiered review logic distinguishing a routine patch bump from a maintainer handoff or a newly introduced install script. Pairing an automated update tool with the verification and review layers described in this article closes that gap.

How much engineering time does reaching Level 2 (Documented) in the maturity model typically require? This varies by the number of language ecosystems and build pipelines in use, but for a single-language, single-pipeline organization, generating and archiving an SBOM per release, plus a basic SBOM diff check, is commonly achievable within one or two sprints using existing open-source tooling (syft, cyclonedx-npm, or ecosystem-native equivalents), without new infrastructure procurement.

Is dependency provenance verification relevant to internal-only tools, or just customer-facing production software? It's relevant to both, and internal tooling is a genuinely underappreciated exposure — the dependency confusion research cited earlier compromised internal build systems specifically, not customer-facing applications, and internal tools frequently run with broader infrastructure access and weaker review scrutiny than production-facing code.

Do reproducible builds make signed provenance attestation unnecessary? No — they're complementary, not substitutes. A signed attestation tells you who or what produced an artifact and when. Reproducibility lets an independent party confirm that the same source and build inputs produce the identical artifact, which means you're not solely dependent on trusting the original signer's infrastructure. Most organizations will find signed attestation achievable well before full build reproducibility, and should pursue it first.

Should container images be treated as a separate problem from application dependencies? No, and treating them separately is a common gap. A base image referenced by a mutable tag carries exactly the same tampering exposure described in the CI Actions section — the fix (pin by content digest rather than tag) is mechanically identical, and cosign's image-signing support means the SBOM and signature-verification practices described for application dependencies extend directly to container images without requiring separate tooling.

If our organization is too small to implement the full maturity model, what's the single highest-leverage first step? Confirm that CI actually installs from a committed lockfile with strict, hash-verified installation (npm ci, pip install --require-hashes, or the ecosystem equivalent) rather than a looser install command that silently re-resolves versions. It costs close to nothing, requires no new tooling in most cases, and closes off a meaningful share of the tampering and substitution risk described throughout this article before any budget conversation is needed.