From 422 Security Findings to Auditable Evidence: Building Trust in a DevSecOps Pipeline

Part 2 — How vulnerability triage, SBOMs, provenance, cryptographic signatures, and independent validation changed my approach to software supply chain security.

Introduction

In Part 1 — 12 Green CI Jobs, 422 Security Findings: Lessons from Building an Observability Platform, I shared a situation that changed how I evaluate continuous integration pipelines.

While developing InfraPulse, a monitoring and observability platform I'm building as a hands-on engineering and technical portfolio project, I reached an important milestone: 12 successful GitHub Actions jobs.

The pipeline covered code quality, unit testing, PostgreSQL integration, API contracts, end-to-end testing, observability, Docker image validation, and foundational security controls.

Everything looked promising.

Then I introduced dedicated vulnerability scanning.

The scanners reported 422 findings, including one classified as critical and 95 as high severity.

Those findings did not represent 422 unique or necessarily exploitable vulnerabilities. They were occurrences reported across different components and scanning contexts, requiring further investigation.

The central lesson was:

A green pipeline is evidence of engineering quality. It is not a certificate of zero risk.

But that discovery raised another question:

How can we transform scanner results and automated checks into evidence that can be verified, correlated, and trusted?

That question guided the next stage of InfraPulse's development.

1. The Problem Doesn't End When a Scanner Finds a Vulnerability

One of the first challenges in vulnerability management is understanding what scanner results actually mean.

During the initial InfraPulse security audit, I used three tools:

• pip-audit for Python dependencies.
• npm audit for frontend dependencies.
• Trivy for container images.

The consolidated results were:

The severity distribution included one critical occurrence, 95 high, 143 medium, 15 moderate, 159 low, and nine unknown.

These numbers were significant, but they were not enough to establish the actual risk to the application.

The same advisory can appear in multiple container images. A vulnerable package may exist only during the build process. A Linux distribution may backport a security fix without changing the upstream version number reported by a scanner.

Furthermore, identifying a vulnerable component does not automatically demonstrate that the affected functionality is reachable or exploitable.

From Discovery to Triage

Turning a finding into an engineering decision requires answering several questions:

1. Which component was identified, and in which artifact?
2. Is the component present in the production runtime?
3. Does the vulnerability apply to the version and configuration being used?
4. Is a compatible fix available?
5. Does the same underlying issue appear across multiple targets?
6. What is the potential impact, and how should remediation be prioritized?

This is the difference between automated vulnerability detection and vulnerability management.

In InfraPulse, the initial vulnerability audit remained informational.

This allowed the project to preserve scanner results and investigate their quality before converting them into blocking CI policies.

It did not mean that critical findings were considered acceptable.

It meant recognizing that a severity label alone is not a complete security policy.

An important limitation remains: the full triage and deduplication of the original 422 findings have not yet been completed.

Therefore, I cannot claim that all findings have been classified, remediated, or formally accepted.

2. Security Policies Must Distinguish Different Failure Conditions

One important engineering decision was separating the operational result of a scanner from the outcome of its analysis.

Consider four possible states:

This distinction prevents a dangerous failure mode: interpreting the absence of a vulnerability report as evidence that no vulnerabilities exist.

Imagine that Trivy fails because of an execution error.

If the pipeline converts that failure into an empty findings list, the result may appear secure even though the scan never completed successfully.

That would be a fail-open behavior.

For mandatory security controls, an inability to obtain or validate required evidence should not silently become a successful result.

However, this does not mean every reported vulnerability must immediately block every delivery.

A reliable blocking policy needs explicit criteria for applicability, priority, evidence quality, remediation, and exceptions.

In InfraPulse, foundational integrity and execution controls became mandatory, while vulnerability discovery remained informational during this development phase.

The final coverage and completeness policy is still under development.

Running a security scanner and designing a trustworthy policy around its results are two different engineering problems.

3. SBOM: First, We Need to Know What We're Analyzing

The next stage introduced a Software Bill of Materials (SBOM).

An SBOM is a structured inventory of the components associated with a software artifact.

It helps answer a seemingly simple question:

What components and dependencies are associated with the software we're building?

In InfraPulse, I implemented CycloneDX SBOM generation covering:

• Python dependencies.
• npm dependencies.
• The backend Docker image.
• The Nginx Docker image.

Dependency inventories use lockfile-based inputs, while container images are analyzed through image inspection and scanning tools.

The implementation also introduced validation tests for the structure and consistency of generated documents.

What Does an SBOM Actually Prove?

An SBOM provides visibility into the components identified by its generation process.

But there is an important distinction:

A structurally valid SBOM is not necessarily a complete inventory of the artifact.

If a scanner fails to recognize a package, the resulting document may still be valid while omitting that component.

Similarly, comparing two results generated by the same scanner may demonstrate consistency without establishing independent completeness.

This became one of the central concerns of Sprint 06.

The container image inventories therefore remain classified as COVERAGE_PARTIAL.

Producing a valid CycloneDX document is only the beginning.

We also need to understand which files, layers, and components were actually examined — and which remain outside the verified scope.

4. Integrity Is Not Authenticity: Introducing Provenance

Once the project could generate component inventories, another question emerged:

How can we verify that an SBOM corresponds to the material it is supposed to represent?

This led to the introduction of provenance mechanisms.

InfraPulse now produces structured documents using in-toto Statement v1 and the SLSA Provenance v1 predicate format.

These documents represent information about artifacts, materials, and the processes associated with their production.

The local implementation validates properties such as:

• File and inventory integrity.
• Consistency between expected and observed materials.
• Content hashes.
• Docker image identifiers.
• Component relationships.
• Consistency between SBOMs and provenance evidence.

One important design concern was avoiding blind trust in identifiers supplied by the artifact being verified.

For example, when a Docker image reports an identifier, validation should compare it against an independently established expectation.

Otherwise, an altered artifact could present an equally altered identifier and appear internally consistent.

Two Concepts That Must Not Be Confused

Integrity: Does the content match what was established as expected?

Authenticity: Is there trustworthy evidence about who produced or signed that content?

A hash can demonstrate that content matches an expected digest.

However, if an attacker can replace both the artifact and its reference hash, that comparison alone does not establish trust in the artifact's origin.

Likewise, producing a document that follows the SLSA provenance structure does not automatically establish compliance with a particular SLSA level.

InfraPulse does not claim a specific SLSA level or independently established build-producer authenticity.

FIGURE 1 — CycloneDX SBOM and Provenance Evidence

Figure 1A — CycloneDX SBOM

Figure 1A. Real SBOM evidence from InfraPulse Sprint 06. A CycloneDX 1.7 document generated by Trivy 0.76.1, containing 50 component entries from the Python dependency inventory.

Figure 1B — in-toto/SLSA Provenance

Figure 1B. Real provenance evidence from InfraPulse Sprint 06. An in-toto Statement v1 with a SLSA Provenance v1 predicate, a SHA-256 reference to python.cdx.json, and a declared COVERAGE_PASS status. The local-untrusted context does not independently establish producer authenticity. The coverage status applies specifically to the Python inventory, not to the container images.

Together, Figures 1A and 1B demonstrate the relationship between a generated software component inventory and the structured provenance metadata describing its production.

However, a valid CycloneDX SBOM and a structurally correct provenance statement do not automatically establish complete artifact coverage or trusted build provenance.

This example illustrates why integrity, authenticity, and completeness must be evaluated separately.

5. Cryptographic Signatures and Trust in the Signer

After establishing integrity validation, the next step was cryptographic signing.

For this, I used Cosign, a tool from the Sigstore ecosystem.

The implementation uses asymmetric cryptographic signatures and offline verification, with a previously established trusted public-key reference.

The validation model distinguishes three properties:

These properties answer different questions.

A mathematically valid signature does not necessarily mean the signer is trusted.

If a verifier accepts a public key supplied exclusively by the same package being verified, an attacker could generate a new key pair and sign modified content.

The signature would be valid, but trust would not be established.

That is why InfraPulse uses an independent trusted reference for the authorized public key, including fingerprint verification.

Negative tests were also introduced to reject altered content, tampered bundles, unauthorized keys, and material inconsistencies.

FIGURE 2 — Integrity, Signature, and Trust

Figure 2. Conceptual evidence-validation model in InfraPulse: content integrity, cryptographic signature validity, and trust in the authorized public key. These checks were tested with disposable keys and do not represent operational production-build authentication.

What Was Actually Validated?

The implementation was tested with disposable cryptographic keys, including an integrated execution on a GitHub-hosted runner.

That environment performed signing and verification operations, negative tests, and validation of previously generated evidence.

The tested workflow completed successfully.

However, a disposable test key is not an operational production identity.

InfraPulse does not yet have an operational signing process that establishes production-build authenticity through a trusted institutional signer.

The tests demonstrated that the mechanisms work under the evaluated conditions.

They did not establish that the entire production supply chain is authenticated.

6. Turning CI into an Evidence Producer

By this stage, the pipeline was already executing tests, generating inventories, and validating materials.

But there was an opportunity to improve traceability.

Instead of producing only a green status, CI could publish structured evidence describing what it actually executed.

This idea guided the implementation of a dedicated evidence-generation stage.

The ci-evidence (unsigned) job began producing artifacts containing:

• SBOMs.
• Provenance statements.
• Component manifests.
• Hashes and file sizes.
• Execution references.
• Information about analyzed materials.
• Known coverage limitations.

The architecture deliberately separates two responsibilities.

Evidence production: performed by CI without operational signing keys.

Signing and trust establishment: performed separately, under its own controls.

This separation reduces the need to expose sensitive signing credentials to the workflow that builds and tests the software.

It also prevents signatures from becoming an automatic property of any artifact a CI job can produce.

FIGURE 3 — CI Evidence Production and Validation Flow

*Figure 3

. Conceptual flow of evidence generation, independent validation, and offline signing in InfraPulse. Signing is separate from the evidence-producing CI workflow and was validated with disposable keys rather than operational production credentials.*

Validating Evidence Outside the Producer

During the next phase, I used a disposable GitHub-hosted runner to execute an integrated validation.

The process retrieved evidence from a specific CI execution, verified materials, processed OCI files, imported images into a disposable environment, and performed signing and verification with temporary keys.

It also executed negative tests designed to detect tampering and unauthorized identities.

The integrated validation completed successfully.

This demonstrated that the verification workflow could run outside the environment that originally produced the evidence.

However, independent execution does not automatically prove the authenticity of the original producer.

Container image inventory coverage also remains partial.

The value of the process is making security claims verifiable while keeping their limitations explicit.

FIGURE 4 — Real GitHub Actions Execution

Figure 4. Real InfraPulse GitHub Actions execution showing successful unsigned evidence generation (ci-evidence), informational vulnerability auditing (vulnerability-audit), and the mandatory aggregate gate (ci-required). The workflow completed successfully and published 10 artifacts. This result does not imply zero vulnerabilities or complete container-image coverage.

The screenshot provides evidence of a specific successful CI execution.

It should not be interpreted as proof that all vulnerability findings have been remediated or that every generated artifact has complete coverage.

7. The Next Challenge: Proving OCI Image Coverage

One of the most technically challenging problems emerged while evaluating Docker image inventories.

An OCI image consists of ordered filesystem layers.

One layer may introduce a file. Another may replace it. A later layer may remove it.

Therefore, simply combining all files found across layers does not necessarily represent the image's final filesystem.

Example: A File Removed by a Later Layer

Consider an image with two layers.

Layer 1

/app/config.json /app/server.py 

Layer 2

/app/.wh.config.json /app/requirements.txt 

The .wh.config.json entry represents a removal mechanism known as a whiteout.

In the final filesystem view, config.json should no longer be present simply because it appeared in the first layer.

This behavior must be interpreted correctly when reconstructing the image's virtual filesystem.

Other cases include opaque directories, hardlinks, symbolic links, path collisions, and the order in which filesystem operations are applied.

FIGURE 5 — OCI Layers and Whiteout Semantics

Figure 5. Conceptual OCI filesystem reconstruction example. A whiteout in a later layer logically removes a file introduced by an earlier layer. Inspecting layers independently does not establish the completeness of the final filesystem inventory.

Why Does This Matter for Security?

An incorrect filesystem inventory can produce incorrect conclusions about which components are actually present.

It may report files that were removed by later layers or fail to interpret the final state of the image correctly.

To address this problem, Sprint 06 introduced:

• A structured coverage-state contract.
• A safe OCI/USTAR layer parser.
• Validation of paths, metadata, hashes, and resource limits.
• Negative tests for invalid inputs and unexpected behavior.

These foundations have been implemented and validated.

However, full virtual filesystem reconstruction and independent inventory reconciliation remain incomplete.

The pending work includes applying whiteout semantics, handling opaque directories and links, and reconciling the reconstructed filesystem against component inventories.

For that reason, the current policy does not automatically promote container images to COVERAGE_COMPLETE.

Acknowledging partial coverage is more trustworthy than claiming completeness that has not been demonstrated.

8. How This Changed My Approach to Security

InfraPulse's evolution reinforced a change in perspective.

Initially, the main question was:

Did the pipeline pass?

Then it became:

Which checks passed, and what does each one actually demonstrate?

Now the question is broader:

Which security claims can we support with verifiable, independent, and sufficiently complete evidence?

These questions are not equivalent.

A pipeline can pass because all mandatory checks executed correctly.

A scanner can report findings without every occurrence representing an exploitable vulnerability.

An SBOM can be valid without fully covering an artifact.

A signature can be cryptographically valid without the signer being authorized.

A provenance statement can have the correct structure without establishing the authenticity of the build environment.

Each property must be evaluated separately.

Five Practical Lessons

1. Don't confuse successful execution with the absence of risk.

A green CI job demonstrates that its acceptance criteria were satisfied. It does not automatically answer every security question.

2. Preserve findings before automating decisions.

Vulnerability findings require context, correlation, classification, and remediation criteria.

3. Separate integrity, authenticity, and completeness.

These properties are related, but none can replace the others.

4. Test failure paths, not only successful execution.

Security tools should reject invalid inputs, tampered evidence, unauthorized identities, and unverifiable conditions.

5. Document what has not yet been proven.

A known and documented limitation supports better engineering decisions than an unsupported claim of complete trust.

9. What Still Remains in InfraPulse

Despite the progress, the software supply chain security work is not finished.

The next engineering activities include:

• Completing whiteout, opaque-directory, and link semantics in virtual OCI filesystem reconstruction.
• Developing inventory verification that is more independent of the primary scanner.
• Reconciling identified components with the reconstructed filesystem.
• Expanding negative coverage tests.
• Establishing final completeness criteria.
• Advancing vulnerability triage and prioritization.
• Designing an operational signing and trust policy appropriate for a real delivery environment.

These are future activities, not completed capabilities.

It is also important to recognize that successful tests with disposable keys do not replace operational key management, rotation, revocation, or signer authorization.

The goal is not to make every security tool report green results at any cost.

The goal is to build a process in which every result has a clear technical meaning, known limitations, and evidence that can be independently examined.

Conclusion

In Part 1, the 422 findings demonstrated that a pipeline with 12 successful jobs could still reveal significant potential security risks.

In Part 2, the challenge evolved.

The focus expanded from detecting problems to building mechanisms for verifying inventory integrity, establishing trust in signatures, preserving CI evidence, and understanding coverage limitations.

Throughout this journey, InfraPulse incorporated CycloneDX SBOMs, provenance structures, cryptographic verification with Cosign, structured CI evidence, and integrated validation in disposable environments.

These mechanisms improved the project's ability to investigate and verify results.

But they did not eliminate every uncertainty.

Container image coverage remains partial. Vulnerability triage still requires further work. Operational build-producer authenticity has not been established.

And that may be the most important lesson from this stage:

Security is not about making every indicator green. It is about knowing exactly what each indicator proves, which risks remain, and what evidence supports our decisions.

A trustworthy pipeline does not need to claim that everything is secure.

It needs to demonstrate what has been verified — and acknowledge what has not.

The journey continues.

There is still much to explore, validate, and improve. I'll continue sharing the engineering challenges, lessons learned, and progress behind InfraPulse.

Until next time!

References and Tools

• GitHub Actions Documentation
• CycloneDX — SBOM Standard
• OWASP Software Component Verification Standard
• Trivy Documentation
• pip-audit — PyPA
• npm audit Documentation
• in-toto — Supply Chain Attestations
• SLSA — Supply-chain Levels for Software Artifacts
• Sigstore Cosign Documentation
• Open Container Initiative — Image Specification