Kubernetes CRDs for ML platforms

A CustomResourceDefinition, or CRD, extends the Kubernetes API with a new resource type. It lets teams represent domain concepts—such as a distributed training job, model rollout, or accelerator claim—as objects that work with familiar API discovery, RBAC, watches, and kubectl workflows.

A CRD stores intent; it does not implement behavior. That distinction is the foundation of a reliable extension.

Separate the API from the controller

The Kubernetes custom resources documentation distinguishes the resource definition from the custom controller that acts on it.

The CRD defines the API group, versions, names, scope, schema, and optional status or scale subresources. A controller watches instances of that resource and reconciles the world toward their desired state.

Without a controller, a TrainingJob object is only structured data in the API. With a controller, it can create worker Pods, observe their progress, retry failures, update status, and clean up dependents.

Treat these as separately versioned contracts. The API must remain understandable to clients while controller releases change implementation behavior.

Use a CRD when the concept is truly declarative

A strong custom resource has a clear desired state and benefits from Kubernetes reconciliation. It often needs to be discovered, watched, authorized, and composed with other cluster resources.

Good candidates include infrastructure or workload concepts with a lifecycle longer than one API request. Weak candidates include high-volume telemetry, application data, and actions that cannot be represented safely as desired state.

Before creating a CRD, ask:

• Can a built-in Job, Deployment, StatefulSet, or configuration object express the requirement?
• Does the resource need Kubernetes API discovery, RBAC, admission, and watch semantics?
• Is there a controller team responsible for reconciliation, upgrades, and incidents?
• Will objects remain bounded in size and count?
• Can users understand the status without reading controller logs?

Adding a CRD creates a long-lived API surface, not merely a new YAML template.

Design a stable schema

Use structural OpenAPI validation and make invalid states difficult to express. Choose a namespaced resource unless the concept genuinely spans the entire cluster. Keep user intent in spec and controller observations in status.

Status should include conditions with clear types, reasons, messages, and transition times. Conditions such as Accepted, Scheduled, Running, and Succeeded are more useful than a single opaque phase when several controllers participate.

Plan API evolution before the first production version. A CRD can serve several versions, but one storage version persists in etcd. Incompatible schema changes require conversion and migration planning. Deprecate fields deliberately and document defaults rather than relying on controller guesses.

Make reconciliation idempotent

A controller can receive the same event repeatedly, restart midway through work, or observe dependencies in an unexpected order. Its reconciliation must be idempotent: computing the desired next state twice should not create two external resources or corrupt ownership.

Use owner references for Kubernetes dependents and finalizers only when external cleanup is necessary. A finalizer is a promise that a controller will eventually remove it. If that controller is unavailable or incompatible, deletion can remain blocked.

Record failures in status and emit bounded events. Retry transient errors with backoff, but surface invalid configuration without endlessly repeating a request that cannot succeed.

Secure the extension

A custom resource can indirectly create privileged Pods, cloud resources, network exposure, or credentials. Its authorization model must cover both the CRD and everything the controller can create.

Grant the controller the minimum RBAC needed. Validate security-sensitive fields through schema and admission policy. Avoid allowing arbitrary Pod templates unless every inherited field is reviewed. Separate user permissions to create a custom resource from administrator permissions to install, update, or delete its definition.

Audit changes to CRD instances and the definition itself. A schema or controller upgrade can affect every existing resource in the cluster.

Operate CRDs as platform dependencies

Inventory the CRDs that an ML platform depends on, including their owners, controller versions, supported Kubernetes versions, upgrade order, metrics, and recovery process. Monitor reconciliation errors, queue depth, work duration, API throttling, and objects stuck on old generations.

Compare metadata.generation with an observed generation in status. A controller that reports healthy conditions for an older generation has not yet processed the user's latest intent.

Use explicit context and namespace when inspecting custom resources:

kubectl --context ml-production --namespace team-a get trainingjobs.platform.example.io
kubectl --context ml-production --namespace team-a describe trainingjob.platform.example.io run-42

Fit CRDs into Polyaxon deliberately

Polyaxon operations provide a consistent ML workflow across Kubernetes primitives and supported integrations. Custom resources may still be appropriate underneath the platform for distributed training, device management, networking, or infrastructure automation.

Keep the user-facing contract at the Polyaxon layer when portability and reproducibility matter. Use CRDs as implementation integrations with explicit compatibility, permissions, and ownership. Polyaxon scheduling presets and node scheduling can centralize policy that should not be copied into every workload.

The best CRD is not the most flexible one. It is the smallest durable API that expresses a real domain concept, reports its state clearly, and has a controller the platform team is prepared to operate for its full lifetime.