Enforce ML workload rules with CEL admission policies

A platform defines an ownership label and an approved image registry for ML workloads. Templates help teams follow those conventions, but a manually submitted Pod or another controller can still send a different specification to the cluster.

Kubernetes ValidatingAdmissionPolicy evaluates CEL expressions inside the API server's admission path. It has been stable since Kubernetes 1.30. A policy defines the checks; a binding selects where and how they apply. ValidatingAdmissionPolicy documentation.

This walkthrough adds a concrete policy to the broader admission-controller guide: require an owner label and digest-pinned images from an approved repository prefix when Pods are created in selected namespaces.

Define a small, inspectable contract

Use rules that a workload author can understand and fix. In this example:

• ml.example.com/owner must be present and nonempty.
• Regular and init containers must use images under registry.example.com/ml/.
• Those image references must end in a SHA-256 digest.

The domain and registry prefix are examples to replace with your organization's values. A digest pins content identity; checking its text does not verify an image signature, scan its contents, or authorize the person named in the owner label.

The policy is intentionally limited to Pod creation. It does not cover later image updates or the ephemeralcontainers subresource. Those need separately designed rules if they are part of your enforcement boundary.

Write the validation policy

Use Kubernetes 1.30 or later and a cluster-administrator-managed trial namespace. Creating these cluster-scoped policy resources requires appropriate permissions. Coordinate the namespace selection before applying them.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: ml-pod-contract.example.com
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        has(object.metadata.labels) &&
        'ml.example.com/owner' in object.metadata.labels &&
        object.metadata.labels['ml.example.com/owner'] != ''
      message: "Set a nonempty ml.example.com/owner label on the Pod."
    - expression: >-
        object.spec.containers.all(c,
          c.image.startsWith('registry.example.com/ml/') &&
          c.image.matches('@sha256:[0-9a-f]{64}$'))
      message: "Use approved registry.example.com/ml/ images pinned by SHA-256 digest."
    - expression: >-
        !has(object.spec.initContainers) ||
        object.spec.initContainers.all(c,
          c.image.startsWith('registry.example.com/ml/') &&
          c.image.matches('@sha256:[0-9a-f]{64}$'))
      message: "Use approved digest-pinned images for init containers too."

The optional init-container field is guarded before it is traversed. Every declared regular container is checked, so a compliant main container does not hide an unapproved sidecar. The policy API reference describes the matched resources and validation expressions.

Keep this contract aligned with your actual image supply process. If approved images are mirrored under several prefixes, model those approved choices deliberately rather than weakening the rule until any image passes.

Bind it in warning mode first

This binding selects namespaces labeled policy.example.com/ml: "true". Have the platform owner apply that label only to the trial namespace initially.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: ml-pod-contract-trial.example.com
spec:
  policyName: ml-pod-contract.example.com
  validationActions: [Warn, Audit]
  matchResources:
    namespaceSelector:
      matchLabels:
        policy.example.com/ml: "true"

Warn reports failed checks to the request client; Audit annotates the request's audit event. These actions permit the request. Audit evidence is useful only when the cluster's audit configuration and log access make it available. After reviewing the trial, change the binding to [Deny, Audit] to reject violations. Deny and Warn cannot be combined. Binding API.

Control who can change the namespace label. If any workload author can remove it, namespace selection becomes an opt-out rather than a platform enforcement boundary.

Review controller-generated Pods too

Inspect the policy's status.typeChecking.expressionWarnings before expanding the rollout:

kubectl get validatingadmissionpolicy ml-pod-contract.example.com -o yaml

Then exercise the intended cases in the trial environment. The following is a review matrix, not a record of executed results:

Include Pods produced by Jobs, Deployments, notebooks, and inference controllers. An accepted Job object does not prove its future Pods will pass this policy. If a controller's Pod creation is rejected, its events and logs may hold the useful message rather than the user's original API response.

Also include admission-time sidecar injection in the trial. Injected containers must satisfy the same contract when this validation runs. Coordinate approved images with the owners of those injectors before enforcing the rule.

Pair Polyaxon defaults with cluster enforcement

Polyaxon scheduling presets make the approved configuration reusable. A platform team can package workload labels, resource settings, and the required environment configuration so training, notebook, and inference users start from a maintained profile. In Polyaxon EE and Cloud, organization and project defaults can apply saved presets without asking every author to repeat the same settings.

Admission policy then checks the generated Pod at the cluster boundary, including submissions from other tools. This gives the team two useful controls: Polyaxon simplifies producing the intended configuration, and Kubernetes verifies that the submitted Pod satisfies the policy. Check the actual Pod labels and all container images during the warning phase, including auxiliary containers; a run or parent Job label must reach the Pod to satisfy a Pod-level rule.

Keep failure messages actionable and ownership clear: the platform team maintains the policy and approved image paths; workload teams fix their inputs. If rolling back the trial, remove its binding first, then remove the example policy after confirming no other binding uses it. Expand enforcement only after representative training, notebook, and inference workloads satisfy the agreed contract.