Kubernetes Pods for ML workloads

A Pod is the smallest deployable unit Kubernetes schedules. It wraps one or more containers that share a network namespace, an IP address, and declared storage volumes.

For ML workloads, a Pod may run a training process, notebook, data-preparation task, evaluator, model server, or one member of a distributed job. It is an execution envelope, not a durable experiment record.

Treat Pods as replaceable

The Kubernetes Pods documentation emphasizes that Pods are relatively ephemeral. A controller creates a replacement when a managed Pod fails, but the replacement has a new UID and can run on another node.

Do not store irreplaceable outputs only in the container filesystem or depend on a Pod name as a permanent identity. Persist checkpoints, models, reports, and other durable artifacts outside the Pod. Track the logical run separately from any individual execution attempt.

A replacement Pod is not a process restart. Local files, IP address, node placement, and runtime state may all change.

Choose the owning workload controller

Create Pods through the abstraction that matches the work:

Avoid bare Pods for production work because no higher-level controller will restore or roll them out. The owner also determines how completion, retries, scaling, and cleanup behave.

Request resources honestly

The scheduler places a Pod using declared requests and constraints, not its future measured consumption. Set CPU, memory, ephemeral storage, and accelerator requests that reflect the workload.

Requests that are too small create contention and eviction risk. Requests that are much larger than real demand waste scarce capacity and increase queue time. Limits add runtime behavior: CPU can be throttled, while memory-limit violations can terminate a container.

For multi-container Pods, remember that regular container requests are generally summed, while init-container calculations follow their own scheduling rules. A helper container is not free simply because it is called a sidecar.

Use Polyaxon scheduling presets to standardize resources and node scheduling to express accelerator, topology, and pool requirements consistently.

Use multiple containers only for one lifecycle

Containers belong in the same Pod when they must be scheduled together and share network or storage lifecycle. Common examples include an initialization step that prepares configuration or a tightly coupled helper that proxies, synchronizes, or exports data.

Do not combine unrelated services merely to reduce object count. Containers in one Pod cannot be placed, scaled, or upgraded independently. A resource or lifecycle problem in one can affect the entire Pod.

Kubernetes now supports native sidecar behavior through init-container semantics in supported versions. Verify cluster compatibility before relying on it. Our Kubernetes sidecars guide covers lifecycle and resource tradeoffs in more detail.

Design probes around real workload state

Startup, readiness, and liveness probes answer different questions:

• startup: has a slow-starting process finished initialization?
• readiness: can this Pod receive traffic now?
• liveness: is the process stuck in a state that restart can repair?

A model server should not become ready before its model is loaded and dependencies are usable. A long training step should not fail liveness merely because it emits no HTTP response. Incorrect probes can turn ordinary initialization into a restart loop.

Use our Kubernetes probes guide to select mechanisms, thresholds, and failure behavior.

Persist data outside the Pod

Use declared volumes for data that must be shared between containers or survive container restarts. Use persistent storage or object storage for data that must survive Pod replacement.

Keep immutable inputs and versioned outputs separate. Define what happens when a run is retried: whether it resumes from a checkpoint, writes to a new attempt path, or replaces an incomplete artifact. Mounting the same path does not make concurrent writes safe.

Polyaxon artifact connections provide controlled access to durable artifact stores without baking credentials into images.

Debug before the Pod disappears

Start with conditions, events, init-container status, and container states:

kubectl --context ml-production --namespace team-a describe pod training-run-42-worker-0
kubectl --context ml-production --namespace team-a logs training-run-42-worker-0 --all-containers=true

Determine whether the failure occurred during admission, scheduling, sandbox creation, image pulling, initialization, application execution, or termination. Capture the node, image digest, exit reason, restart count, and owner reference.

Polyaxon run logging keeps workload output connected to the logical operation. Kubernetes evidence explains the execution attempt; Polyaxon explains why it ran and which inputs and outputs belong to it.

Pods are intentionally replaceable. Design the ML system so replacement is routine: controllers recreate execution, external systems preserve durable state, and the platform retains the context required to reproduce and understand the work.