Container image scanning for ML workloads
Build container image scanning into ML delivery with digest pinning, SBOMs, provenance, policy, remediation ownership, and runtime controls.
Container image scanning identifies known vulnerabilities, exposed secrets, license concerns, and selected configuration risks before an image runs. It is one control in a supply-chain program, not proof that an image is safe.
ML images make the problem harder. They can contain large operating-system layers, Python or JVM dependencies, GPU runtimes, compilers, notebooks, model servers, and tools copied from development environments. A scan must lead to ownership and remediation rather than become another report nobody reads.
Know what the scanner evaluates
Most scanners inspect operating-system packages and language dependencies against vulnerability databases. Some also detect secrets, malware patterns, licenses, Dockerfile issues, or Kubernetes configuration.
Coverage depends on the image contents and available package metadata. A scanner may miss:
- software compiled from source without discoverable version information;
- malicious application logic;
- runtime-only configuration and mounted dependencies;
- vulnerable services reached over the network;
- unsafe model serialization or poisoned model artifacts;
- newly disclosed vulnerabilities not yet in the database.
Document the scanner's scope and feed freshness. Treat “no findings” as one signal, not an assurance statement.
Scan the exact immutable image
Kubernetes image names can use tags or digests. The official Kubernetes image documentation explains that tags can move while digests identify immutable image content.
Build once, scan the resulting digest, promote that digest, and deploy the same digest. Rebuilding or retagging between stages can invalidate the scan result.
Record the registry, repository, digest, build source, base image, builder identity, scan policy, and timestamp. If a vulnerability database changes later, rescan deployed digests without rebuilding them so operators can distinguish new knowledge from changed software.
Generate and retain an SBOM
A software bill of materials inventories components and versions inside the image. Generate it during the build, associate it with the digest, and store it where security and platform teams can query it.
An SBOM improves investigation when a new vulnerability is disclosed: teams can identify affected images without pulling and rescanning every artifact first. It also supports license review and base-image lifecycle management.
Protect SBOM access. Package names, paths, and internal components can reveal useful information about the software supply chain.
Add provenance and signatures
Scanning asks what is inside the image. Provenance asks how it was built. Signatures help verify that an approved identity attested to an artifact.
Record source revision, build workflow, parameters, builder, dependencies, and output digest. Use short-lived workload identity for signing and verification rather than shared long-lived keys.
Admission policy can require an approved registry, digest, signature, provenance, and scan result. Keep emergency and exception paths explicit, time-bounded, audited, and owned.
Do not let a scanner sign its own broad approval without independent policy. Separate evidence generation from the decision about whether a workload may run.
Design actionable severity policy
Severity alone is a poor deployment rule. Include exploitability, package reachability, exposure, workload privilege, available fix, compensating controls, environment, and business impact.
Define policy outcomes:
- block before production;
- allow temporarily with an expiry and owner;
- require base-image or dependency remediation;
- monitor because no fix exists;
- accept because the affected component is not present at runtime.
Avoid blocking every development experiment on findings that cannot affect it. Avoid allowing production indefinitely because a scanner generated too many false positives. Tune policy by workload class and risk.
Handle ML-specific image risks
Separate container images from model and data artifacts. A clean image can load an unsafe serialized model or untrusted code from a mounted volume. Scan and validate each artifact type with controls appropriate to its format.
Minimize images by separating build and runtime stages, removing compilers and package caches, and including only required GPU libraries. Pin base images and dependencies, but maintain a recurring update process so pinning does not freeze vulnerabilities indefinitely.
Do not bake credentials, datasets, or production configuration into the image. Use Polyaxon connections to provide approved external access at runtime.
Continue controls at runtime
An approved image can still be deployed with excessive privilege, writable host mounts, dangerous capabilities, or unrestricted network access. Combine scanning with admission policy, least-privilege security contexts, network controls, workload identity, audit logs, and runtime detection.
Inventory deployed digests and compare them with current policy. Alert when an image becomes newly vulnerable, loses support, or no longer has an owner.
Polyaxon makes the execution environment part of the tracked operation, connecting the image to code, parameters, resources, logs, and artifacts. This supports reproducibility and impact analysis when an image must be replaced.
Container image scanning works when it is digest-specific, repeatable, tied to build provenance, and backed by remediation ownership. Use it as one layer in a system that also secures configuration, identity, data, models, and the running workload.