Kubernetes admission controllers for ML platforms

Kubernetes admission runs after authentication and authorization but before an API object is persisted. Admission controllers can default, mutate, validate, or reject create and update requests according to cluster policy.

This is a powerful boundary for ML platforms: it can enforce safe images, resources, identity, placement, and security context before an expensive or privileged workload reaches the scheduler. It is also part of the API availability path, so a poorly designed webhook can block an entire cluster.

Understand the request sequence

The official admission controller documentation places admission after a request is authenticated and authorized.

A simplified write path is:

1. authenticate the caller;
2. authorize the requested verb and resource;
3. run mutating admission;
4. apply the resulting object changes;
5. run validating admission;
6. persist the accepted object.

Admission does not generally intercept read operations. It also does not replace RBAC: authorization decides whether an identity may make the request, while admission decides whether the proposed object satisfies policy.

Built-in admission plugins run within the API server. Dynamic admission can use policies and external webhooks configured through Kubernetes resources.

Prefer the smallest policy mechanism

Use built-in controls first when they solve the requirement. Namespace quota, LimitRange, Pod Security Admission, and other native features have well-understood lifecycle and availability characteristics.

For declarative validation, evaluate ValidatingAdmissionPolicy and its CEL expressions before building an external service. A webhook is justified when policy needs external data, complex logic, custom mutation, or behavior unavailable in simpler mechanisms.

Every new webhook creates a network call and a release lifecycle in the API write path. The benefit must outweigh that operational cost.

Separate mutation from validation

Mutation can add safe defaults such as platform labels, tolerations, sidecars, or standard environment settings. Validation can reject disallowed registries, mutable tags, missing resource requests, excessive privilege, or unsupported scheduler configuration.

Keep mutations predictable and visible. Users should be able to inspect the final object and understand which controller changed it. Avoid silently transforming user intent into materially different resources.

Design validators to accept every valid form, including fields added by other mutators. Webhooks may be called more than once, so mutations must be idempotent: applying the same mutation repeatedly should produce the same object.

Scope webhooks narrowly

Use rules, selectors, and match conditions to limit a webhook to the operations and resources it understands. Exclude the webhook's own namespace and dependencies so operators can recover it during failure.

Avoid matching all Pods across every namespace when the policy applies only to selected ML workloads. Broad scope increases latency, availability risk, and the chance of breaking system components.

The Kubernetes admission webhook good practices recommend consolidating calls, using small timeouts, avoiding dependency loops, and designing for high availability.

Choose failure policy deliberately

failurePolicy: Fail rejects matching requests when the webhook errors or times out. It protects fail-closed security requirements but can stop deployments and recovery work.

failurePolicy: Ignore allows requests to continue when the webhook is unavailable. It protects API availability but may allow resources that were not validated or mutated.

Choose by policy impact. A security boundary may require fail-closed behavior with a highly available webhook. A convenience mutation may be safer fail-open, paired with a validating controller that detects missing configuration later.

Document the degraded mode and how to disable or repair the webhook when it blocks the cluster.

Keep the webhook fast and available

Admission latency adds directly to API write latency. Use short timeouts, avoid sequential external calls, cache bounded reference data, and return clear status messages.

Run multiple replicas with appropriate disruption and topology policy. Monitor request count, latency, timeout, error, rejection reason, certificate expiry, and dependency health.

Avoid calling back into resources that trigger the same webhook or waiting on systems whose recovery requires new Kubernetes objects. Dependency loops can turn a small outage into a control-plane incident.

Enforce ML workload policy

Useful ML admission rules include:

• require immutable image digests and approved registries;
• require CPU, memory, storage, and accelerator requests;
• restrict privileged mode, host mounts, capabilities, and host networking;
• allow only approved service accounts and workload identities;
• validate queue, scheduler, node pool, and topology fields;
• enforce required ownership and cost labels;
• restrict unapproved sidecars and injected agents;
• reject connections or volumes outside the tenant's policy.

Keep model, prompt, or data-quality validation out of the Kubernetes admission path unless it truly belongs to object acceptance. Those checks often need a separate ML evaluation workflow with richer evidence and different failure semantics.

Integrate admission with Polyaxon

Polyaxon scheduling presets provide reusable workload defaults before Kubernetes admission, while RBAC governs access to platform organizations, projects, and resources. Kubernetes admission remains the cluster-wide enforcement boundary for accepted objects.

Return precise rejection messages that name the invalid field, rule, and approved remediation. Track policy versions with the workload so teams can explain why a configuration was accepted previously and rejected later.

Admission policy succeeds when it prevents unsafe states without surprising users or reducing control-plane reliability. Prefer native, declarative mechanisms; scope dynamic webhooks narrowly; and operate every webhook as a critical dependency of the Kubernetes API.