Kubernetes container runtimes for ML workloads

Kubernetes schedules Pods, but a container runtime on each node pulls images, prepares container filesystems, and manages container processes. This layer is usually invisible until image pulls slow down, sandboxes fail, GPU devices are missing, or two node pools run the same workload differently.

ML platform teams do not need to implement a runtime, but they should understand the boundaries that determine portability, security, startup time, and diagnostics.

Separate the runtime layers

Three specifications are easy to conflate:

• The Open Container Initiative image specification defines the format used to distribute container images.
• The OCI runtime specification defines how a container process is configured and executed.
• The Kubernetes Container Runtime Interface (CRI) defines how the kubelet communicates with a node's container runtime.

A CRI implementation such as containerd or CRI-O can use a lower-level OCI runtime to create the process. Kubernetes talks to the CRI endpoint; ordinary workloads should not depend on a vendor-specific runtime CLI.

The current Kubernetes container runtime documentation describes supported CRI integration and node configuration. Docker-built OCI images can run on Kubernetes without the removed dockershim integration; the image format and the node runtime are separate concerns.

Inspect what each node runs

Kubernetes reports the runtime for each node:

kubectl --context production get nodes \
  -o custom-columns='NAME:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion,OS:.status.nodeInfo.osImage,KUBELET:.status.nodeInfo.kubeletVersion'

Compare runtime and operating-system images by node pool. A mixed inventory may be expected during an upgrade, but a persistent outlier can explain workload-specific failures. Record GPU driver and device-plugin versions separately because they are not represented by the runtime string alone.

Node-level tools such as crictl are for authorized operators and should target the CRI endpoint configured on that node. Application teams should begin with Pod status, events, and logs; escalate to node diagnostics only when the evidence crosses that boundary.

Treat images as immutable inputs

Runtime portability begins with the image:

• pin the base image and application image by an approved version or digest;
• build for the destination architecture;
• use a non-root user where the workload permits;
• keep runtime libraries compatible with accelerator drivers;
• separate build-time tools from the final image;
• record the image digest with the run.

Large ML images amplify every pull and unpack inefficiency. Shared layers, small build contexts, multi-stage builds, and deliberate cache policy can reduce startup time without coupling workloads to one node. See Docker build caching for ML workloads for the build side of that problem.

Account for GPU integration

The runtime does not discover and schedule GPUs by itself. Kubernetes node labeling, device plugins or dynamic resource allocation, drivers, and vendor runtime integration work together to expose devices to a container.

When a Pod starts but cannot use its accelerator, compare:

1. the requested Kubernetes resource;
2. device availability and allocation on the node;
3. device-plugin health;
4. driver and user-space library compatibility;
5. container security settings and runtime configuration;
6. the image's supported architecture and accelerator stack.

Do not add privileged mode as a generic fix. It may hide the real integration problem while materially expanding access to the node.

Use RuntimeClass only for a reason

Kubernetes RuntimeClass lets a Pod select a configured runtime handler. Platform teams might use it for sandboxed runtimes, hardware virtualization, or another isolation/performance tradeoff.

A RuntimeClass name is cluster-specific configuration, not a portable application assumption. If selected workloads require one, expose it through a controlled environment preset and validate that the destination cluster supports it. Account for any resource overhead and scheduling constraints associated with the handler.

Monitor runtime effects

Runtime health appears in workload behavior:

• image pull and unpack duration;
• Pod sandbox creation failures;
• container start latency;
• runtime and kubelet errors;
• filesystem bytes and inodes;
• garbage-collection activity;
• container restarts and exit reasons;
• node-specific failure rates.

Correlate these signals with image digest, node image, runtime version, node pool, and operation. Polyaxon provides the run and workload context; cluster observability supplies the node-level evidence.

Upgrade as a complete node stack

Validate upgrades as combinations rather than isolated version numbers. A representative ML test should pull a cold image, mount required storage, access artifact and data connections, allocate an accelerator, run a short computation, upload an artifact, and terminate cleanly.

Roll through a canary node pool before broad replacement. Preserve a rollback path with the previous immutable node image. The runtime is only one layer, but it sits on every workload's critical path; treat changes to it with the same evidence and release discipline as changes to the ML platform itself.