Kubernetes objects for ML platform teams

Kubernetes objects are durable records of intent. A manifest says what should exist; the API server stores that intent; controllers continuously compare it with the cluster's current state and act on differences.

For an ML platform team, this model connects a high-level run to concrete resources such as Jobs, Pods, Services, ConfigMaps, and persistent storage. Reading those objects as evidence makes failures easier to locate without confusing a platform operation with the individual containers that execute it.

Read every object as desired and observed state

The official Kubernetes objects guide describes two central fields:

• spec expresses the desired state supplied by a user or controller;
• status reports the observed state written by Kubernetes components.

The gap between them is often the most useful diagnostic signal. A Deployment may request three replicas while only two are available. A Job may require one completion while reporting a failed Pod. A PVC may request storage while remaining Pending.

Do not treat a successfully accepted manifest as a successfully running workload. Admission, scheduling, image retrieval, initialization, execution, and cleanup happen after the API accepts the object.

Know the common object roles

Different objects own different parts of an ML workload:

The object at the bottom of the stack is not necessarily the source of truth. A controller-owned Pod is replaceable. Change its owner—such as the Job or Deployment—or the platform configuration that generated that owner.

Use metadata as an operating contract

Every object includes metadata such as its name, namespace, UID, labels, annotations, and ownership. These fields are more than decoration.

Labels support selection and stable grouping. Use a bounded label vocabulary for concepts that systems must query: workload class, queue, project, environment, or component. Annotations are better for non-identifying metadata that should not participate in selectors.

Names can be reused after deletion; UIDs cannot. Incident records should therefore capture the namespace, kind, name, and UID when exact identity matters.

Owner references describe dependency. They let garbage collection and diagnostic tools connect a Pod to the Job, ReplicaSet, or controller that created it. Finalizers delay deletion until a responsible controller finishes cleanup. A resource stuck in Terminating may be waiting for that cleanup, not for the container to stop.

Prefer declarative management

Imperative commands are useful for inspection and bounded troubleshooting. Reproducible platform configuration should remain declarative and reviewable.

For example, this command asks the API server to reconcile a complete manifest in an explicit cluster and namespace:

kubectl --context ml-production --namespace team-a apply --filename training-job.yaml

Avoid editing generated Pods or relying on commands that exist only in a terminal history. Store the intended configuration in version control or the platform that owns it. Review field ownership when several tools manage the same object, because a later reconciliation can restore fields that were changed manually.

Debug the reconciliation chain

Follow the object's transitions rather than jumping directly to container logs:

1. Confirm the object exists in the expected context and namespace.
2. Read its status conditions and recent events.
3. Identify the controller and owner references.
4. Check admission, queue, and scheduler decisions before node execution.
5. Inspect the selected node, Pod conditions, init containers, and container states.
6. Correlate the Kubernetes object with application logs, metrics, and persisted outputs.

Use an explicit command when collecting evidence:

kubectl --context ml-production --namespace team-a describe job training-run-42

Events are useful but retained for a limited period. Capture them before retries or cleanup remove the evidence that distinguishes an admission problem from a scheduling or application failure.

Connect objects to Polyaxon operations

Polyaxon provides the ML-level source of truth: project, operation, inputs, parameters, code, environment, resources, scheduling policy, logs, lineage, and artifacts. Kubernetes provides the execution objects and control loops underneath it.

Reusable Polyaxon scheduling presets keep placement policy out of one-off manifests, while platform observability helps operators correlate service health with workload state. The operations quick start shows how execution configuration becomes a tracked run.

Keep both layers visible during an incident. The Polyaxon operation explains what the user requested and which experiment it belongs to. Kubernetes objects explain how the cluster admitted, placed, ran, and terminated that request.

The result is a clean ownership model: users work with reproducible ML operations, platform teams enforce shared policy, and Kubernetes controllers reconcile the low-level objects required to execute the work.