Kubernetes nodes for ML platforms

A Kubernetes node is the machine on which Pods run. For ML infrastructure, nodes are rarely interchangeable: CPU architecture, memory, local storage, accelerator model, network fabric, pricing, zone, and interruption behavior all influence which workloads can run safely.

Kubernetes abstracts machines behind a common Node API, but good scheduling still depends on accurate node metadata and disciplined lifecycle management.

Know the node components

Every worker node needs:

• a kubelet that registers the node and reconciles assigned Pods;
• a container runtime that creates Pod sandboxes and runs containers;
• networking components that implement Pod and Service traffic for the cluster.

Many clusters also run kube-proxy, although some network implementations provide Service handling through another data plane. Device plugins, storage drivers, logging agents, security agents, and GPU operators commonly run as DaemonSets.

The Kubernetes node documentation describes registration, status, heartbeats, and node-controller behavior.

Inspect status and allocatable capacity

Start with a compact inventory:

kubectl --context production get nodes -o wide
kubectl --context production get nodes \
  -o custom-columns='NAME:.metadata.name,READY:.status.conditions[-1].status,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory,GPU:.status.allocatable.nvidia\.com/gpu,RUNTIME:.status.nodeInfo.containerRuntimeVersion'

The condition list is not guaranteed to place Ready last, so use kubectl describe node or a more deliberate query when diagnosing one machine:

kubectl --context production describe node gpu-worker-1

Capacity is the hardware Kubernetes discovers. Allocatable is what remains available to Pods after system reservations and other adjustments. Neither tells you how much is currently free or whether the remaining resources fit a workload's complete shape.

Read node conditions as signals

Important built-in conditions include:

• Ready: whether the node is healthy enough to accept Pods;
• MemoryPressure: available memory crossed a configured threshold;
• DiskPressure: filesystem bytes or inodes are under pressure;
• PIDPressure: too few process identifiers remain;
• NetworkUnavailable: the node network is not correctly configured where the implementation reports it.

Conditions, taints, events, and workload symptoms belong together. A node can report Ready while a GPU plugin is unavailable, a storage zone is unreachable, or one runtime path is failing.

Model heterogeneous capacity

Use well-governed labels for properties the scheduler can rely on: node pool, accelerator family, accelerator memory class, architecture, zone, purchasing model, or compliance boundary. Prefer labels maintained by the platform or a trusted node-feature system rather than user-editable claims.

Then combine:

• resource requests for CPU, memory, and devices;
• node affinity for required hardware or location;
• taints and tolerations to protect specialized pools;
• topology constraints for distribution;
• queues, quotas, and priority for shared capacity.

Our guide to taints and tolerations for ML workloads explains why a toleration alone does not attract a Pod to the intended pool.

Expect node loss

A Pod remains bound to its assigned node until Kubernetes terminates or deletes it. When a node becomes unreachable, controllers may eventually replace managed Pods elsewhere, subject to policy and available capacity. Kubernetes does not move the process memory or node-local files to the replacement.

ML workloads must persist valuable progress outside the node. Use periodic checkpoints, idempotent task boundaries, and artifact storage that survives the failure you are designing for. Test what happens when loss occurs during a checkpoint or output upload.

Replicated services need readiness, disruption budgets where appropriate, and enough spare eligible capacity to replace a failed replica. A declared replica count is not resilience if every replica depends on one zone or one full node pool.

Maintain nodes as replaceable units

Prefer immutable node images and controlled pool rollouts. A common maintenance sequence is:

1. introduce validated replacement capacity;
2. cordon the target node;
3. drain it according to workload and disruption policy;
4. replace or upgrade the node;
5. run representative validation;
6. continue the rollout while monitoring failures and capacity.

Do not drain GPU or training nodes without understanding checkpoint behavior, local volumes, daemon workloads, and jobs that cannot be safely evicted. A maintenance procedure should state its stopping conditions and rollback path.

Observe node contribution to run outcomes

Track saturation, pressure conditions, filesystem capacity, network errors, runtime health, device health, thermal or hardware errors where exposed, and Pod creation latency. Correlate each signal with node image, pool, zone, runtime version, driver stack, and recent changes.

Polyaxon records the execution environment and operation context around ML runs. Use node scheduling and presets to express intended placement, then preserve the actual node and resource evidence for diagnosis.

A healthy ML cluster is not merely a list of Ready nodes. It is a set of known capacity classes whose workloads can be placed, observed, interrupted, and replaced without losing the execution context that makes results trustworthy.