Secure the AI Supply Chain with Model Provenance and an AI SBOM

Tier: Core · Workflow area: AI governance, risk & compliance

This playbook builds the governance controls that make the AI supply chain accountable: model provenance and model cards, an SBOM-for-AI that lists models, datasets, and their dependencies, and scanning that catches known-vulnerable or tampered components before they ship. It maps to OWASP LLM03 (Supply Chain), OWASP Agentic ASI09, and MITRE ATLAS supply-chain techniques, turning "we downloaded a model" into "we know exactly what we run and where it came from."

When to reach for this

• •
  Your org uses third-party models, fine-tunes, adapters, or datasets and cannot currently answer "what models are in production and where did they come from?"
• •
  You need to satisfy a client or auditor asking for AI supply-chain assurance, or an ISO 42001 supplier-management control (pb-48).
• •
  You are adding a new model or dependency and want a provenance gate before it enters the stack.
• •
  Skip a full program only if you use a single, first-party, fully-controlled model with no external components — rare in practice.

The workflow

1. 1.

   Inventory every AI component. Enumerate models (base and fine-tuned), adapters, datasets, embedding models, and the software dependencies of your inference stack. You cannot secure what you have not listed.

2. 2.

   Capture provenance for each model. Record source, version/commit or hash, license, training-data claims, and a model card. Pin models by digest, not by a mutable tag, so you can prove what you ran.

   Copy

   component: base-llm
     source:   modelhub://vendor/model-7b
     digest:   sha256:9f3c...            # pinned, not "latest"
     license:  Apache-2.0
     card:     model_cards/base-llm.md   # intended use, eval, limitations
     signature: verified (publisher key)
   

3. 3.

   Generate an SBOM-for-AI. Extend a standard SBOM format (CycloneDX supports ML/AI components) to include models and datasets alongside code dependencies, so the AI stack appears in the same inventory discipline as everything else.

   Copy

   # dependency + component scan feeding the AI SBOM
   trivy fs --scanners vuln,license --format cyclonedx -o ai-sbom.json ./ai-stack
   

4. 4.

   Scan for known-vulnerable and tampered artifacts. Run dependency scanners against the inference stack, verify model signatures/hashes against the publisher, and flag serialization-format risks (unsafe pickle-based model files) that can execute code on load.

5. 5.

   Gate ingestion. New models pass through the same provenance-and-scan check before promotion — an unsigned, unpinned, or unscanned model does not reach production. Wire this into the CI gate (pb-43).

6. 6.

   Re-verify on every update. A model or dependency update is a new supply-chain event. Re-capture provenance and re-scan; do not trust a component because an earlier version was clean.

Verify before you trust it

The controls are trustworthy when the inventory is complete and each component's provenance is independently checkable. Completeness: reconcile the AI SBOM against what is actually deployed — pull the running model digests from production and confirm each appears in the SBOM with matching hash. A model serving in prod that is absent from the inventory is the exact blind spot this task exists to close. Provenance: for a sample of components, independently re-verify the signature or hash against the publisher and confirm the license and model card are real, not placeholders. Ingestion gate: attempt to promote an unsigned or unpinned test model and confirm the gate blocks it — a gate that has never rejected anything is unproven.

Pitfalls

• Mutable version references. Pulling a model by tag means the weights can change under you and you cannot prove what ran during an incident. Pin by digest.
• Unsafe serialization formats. Pickle-based model files can execute arbitrary code on load. Prefer safe formats and scan/quarantine untrusted model files before loading.
• Inventorying code but not models and data. An SBOM that lists Python packages but omits the model and its training data misses the AI-specific attack surface. Include models and datasets.
• One-time provenance. Capturing provenance at first ingestion and never re-checking on update leaves you trusting components you no longer verified. Re-verify on change.
• Trusting a model card at face value. Cards are publisher claims, not proof. Treat training-data and eval assertions as claims to corroborate, not guarantees.

Tools

• Model registry with model cardsthe system of record for provenance: source, digest, license, intended use, and evaluation per model.
• AI SBOM (CycloneDX ML/AI)the standard inventory format extended to models and datasets so AI components share the same governance as code.
• Dependency and artifact scanners (Trivy / Grype)catch known-vulnerable dependencies and verify component integrity before promotion.

Client-safe variant

For a client, build the AI SBOM and provenance records inside their environment as their auditable inventory, and integrate with any existing SBOM and supplier-management process rather than standing up a parallel one. Keep model artifacts, hashes, and scan results within their boundary, and deliver the ingestion gate as a control their platform team owns and re-runs on every model update. Map the program to their ISO 42001 supplier control (pb-48) and, for Indian clients, to the security-safeguard expectations under DPDP (pb-52). Do not move a client's models, datasets, or SBOM out of their environment, and do not reuse one client's provenance records or scan baselines in another engagement.