What Kubernetes means for ML teams

Kubernetes is an open-source platform for managing containerized workloads and services through a declarative API. You submit objects that describe desired state, and control loops continuously work to bring the cluster's actual state closer to that declaration.

For ML teams, that makes Kubernetes a strong execution substrate. It is not, by itself, an experiment tracker, feature store, data catalog, model registry, or complete developer platform.

Think in desired state

A Kubernetes object normally has a spec, which describes intent, and a status, which reports what the system observes. A Deployment might declare three replicas; controllers create and replace Pods until three eligible instances are available.

This reconciliation model changes how operators work. They edit the declaring object and let its controller converge. Manually repairing one generated Pod is temporary because the controller still follows the original template.

The official Kubernetes object model calls these objects records of intent. Version-controlled manifests, policy, and platform APIs build on that idea.

Separate the control plane and worker nodes

The control plane exposes the API, stores cluster state, schedules Pods, and runs controllers. Worker nodes run kubelet, a container runtime, networking components, and the Pods assigned to them.

Several core components cooperate:

• the API server validates and serves Kubernetes resources;
• etcd stores API data;
• the scheduler assigns unscheduled Pods to eligible nodes;
• controllers reconcile resources such as Deployments and Jobs;
• kubelet makes the assigned Pod specification run on a node;
• the container runtime starts and stops containers.

This separation helps troubleshoot failures. A Pending Pod is usually a placement or prerequisite problem. A scheduled Pod that cannot start may have image, volume, configuration, or runtime problems. A running Pod can still contain an unhealthy application.

Use controllers instead of managing Pods

A Pod is Kubernetes' smallest deployable compute unit. It contains one or more tightly coupled containers that share networking and can share volumes. Pods are disposable; higher-level workload objects normally create and replace them.

Choose the controller by lifecycle:

An ML platform often generates these objects from a higher-level operation. The owning layer should manage retries, distributed roles, metadata, artifacts, and user-facing status while Kubernetes manages Pods and infrastructure placement.

Understand scheduling inputs

The scheduler filters and ranks nodes using the Pod's declared requirements: CPU and memory requests, extended resources such as GPUs, node selectors, affinity, topology, taints and tolerations, storage topology, and policy.

It does not schedule from a Python process's future needs or the utilization graph you saw yesterday. If a training container omits or understates a memory request, the scheduler receives incomplete information. If it requests an unavailable GPU resource, it remains Pending even when other nodes have free CPU.

Polyaxon scheduling presets help platform teams package approved node, resource, priority, and policy choices for workload authors.

Treat networking and storage as separate systems

Every Pod receives a network identity within the cluster model. Services provide a stable discovery and routing abstraction over selected Pods. Network policies can restrict allowed connections when the cluster's network implementation enforces them.

Volumes make data available inside Pods. PersistentVolumes and PersistentVolumeClaims separate requested storage from a Pod's lifecycle, but durability, topology, performance, snapshots, and recovery still depend on the storage system and driver.

For ML, large datasets should not automatically be copied into container images or mutable node directories. Choose object storage, shared filesystems, block volumes, caches, and local scratch space according to access pattern and recovery needs.

Build security from identities and policy

Namespaces organize namespaced resources but are not a complete security boundary. ServiceAccounts identify workloads to the Kubernetes API. RBAC controls API actions. Network policy, admission policy, Pod security controls, secrets management, image provenance, and node isolation address other boundaries.

Grant the generated workload only the permissions it needs. Anyone who can create a Pod in a namespace may be able to mount Secrets available there, so “cannot read Secrets directly” is not sufficient isolation.

Know what Kubernetes does not provide

Kubernetes supplies general primitives. It does not build source code, choose a CI/CD process, provide databases or storage systems as built-in services, or mandate a logging and monitoring stack. The Kubernetes overview makes these boundaries explicit.

ML teams still need:

• reproducible component and operation definitions;
• experiment parameters, lineage, metrics, and artifacts;
• queues, fairness, and accelerator-aware policies;
• distributed workload coordination;
• notebook and service lifecycle;
• promotion, governance, and audit context.

Put Polyaxon above Kubernetes

Polyaxon turns Kubernetes primitives into an ML-oriented workflow. A user defines an operation with a component, parameters, connections, resources, and execution policy. Polyaxon compiles that intent into the eligible Kubernetes resources and preserves the experiment context around them.

Kubernetes remains visible where it matters: platform teams control clusters, namespaces, nodes, identity, networking, storage, and policy. Practitioners work with projects, runs, components, artifacts, metrics, and services instead of reconstructing that context from ephemeral Pod names.

The useful mental model is simple: containers package a process, Kubernetes reconciles shared infrastructure, and Polyaxon coordinates the ML work performed on that infrastructure.