Kubernetes is a container orchestration system. That description is accurate and uselessly broad. For ML teams, the useful version is simpler: Kubernetes gives you a way to run jobs and services across a pool of machines while a control plane keeps the cluster close to the desired state.

Understanding the architecture matters because training jobs, notebooks, model services, and distributed workloads fail in different layers of the cluster.

Control plane

The control plane manages cluster state. The API server receives requests, the scheduler places pods on nodes, controllers reconcile desired and current state, and etcd stores cluster data.

When the control plane is unhealthy, workload behavior gets weird: pods stop scheduling, status updates lag, controllers stop reconciling, and debugging output becomes stale.

Worker nodes

Worker nodes run the actual workloads. Each node has a kubelet, a container runtime, networking, and local resources such as CPU, memory, ephemeral storage, and sometimes GPUs.

For ML workloads, node shape matters. A preprocessing job may need memory and storage throughput. A training job may need GPUs and high network bandwidth. A model service may need predictable CPU and latency. Kubernetes can schedule all of these, but only if the workload declares its needs clearly.

Pods and services

A pod is the smallest deployable unit in Kubernetes. It may hold one container or a small group of tightly coupled containers. Services provide stable networking for pods, which are otherwise disposable.

Jobs, notebooks, TensorBoards, and model APIs all end up as pods. The difference is lifecycle. A training job should finish. A service should keep running. A distributed workload needs multiple pods that coordinate correctly.

Config and secrets

ConfigMaps and Secrets inject configuration into workloads. They are useful, but they are also easy to abuse. Hardcoding credentials into images or manifests is still a bad idea, even if the YAML deploys successfully.

ML workloads usually need access to data stores, artifact stores, registries, and tracking backends. Those connections should be managed deliberately, not copied into random manifests.

Storage

Kubernetes volumes let pods access persistent or ephemeral storage. The correct storage choice depends on the workload: temporary scratch space, shared training data, model artifacts, checkpoints, or database state.

Stateful ML systems need explicit storage policy. Otherwise teams learn the hard way that a restarted pod is not a backup strategy.

Polyaxon on top of Kubernetes

Polyaxon uses Kubernetes as the execution layer and adds ML-specific abstractions for jobs, services, distributed training, artifacts, queues, presets, and tracking. That lets teams keep the scalability of Kubernetes without making every data scientist become a cluster operator.