Kubernetes components for ML platform teams
Follow an ML workload through the API server, scheduler, controllers, kubelet, runtime, networking, storage, and platform add-ons.
Kubernetes is a collection of cooperating control loops, not one scheduler process that “runs containers.” Understanding which component owns each decision makes ML workload failures much easier to localize.
Follow one operation from desired state to a running Pod, then use the same path in reverse when something goes wrong.
Start with the control plane
The official Kubernetes components overview identifies the main control-plane components:
| Component | Responsibility |
|---|---|
| API server | Exposes the Kubernetes API and processes authenticated, authorized, admitted requests |
| etcd | Stores Kubernetes API state consistently |
| Scheduler | Selects a node for unscheduled Pods |
| Controller manager | Runs controllers that reconcile actual state toward desired state |
| Cloud controller manager | Integrates supported cloud resources and control loops |
In managed Kubernetes, the provider usually operates these components. Platform teams still need their service health, audit events, upgrade behavior, quotas, and incident boundaries.
The API server is the front door
Clients and controllers interact through the API server. A Polyaxon operation results in Kubernetes objects submitted through that API. Authentication establishes identity, authorization checks permission, admission can validate or mutate the object, and persistence records accepted state.
An admission rejection is different from a scheduling failure. The Pod never becomes eligible for scheduling if the API server rejects it. Preserve the exact status reason and responsible policy instead of treating every failed launch as a generic platform error.
Kubernetes audit logs provide request-level evidence at this boundary.
Controllers create and reconcile
A Deployment controller manages ReplicaSets; a Job controller manages finite work; other operators reconcile custom resources. Controllers watch current state and make changes toward declared state.
This explains why editing a generated Pod is often temporary. The owning controller may replace it or restore fields from the higher-level object. Identify ownerReferences and change the source of truth.
ML platforms add controllers for workflows, distributed jobs, devices, serving, and infrastructure. Every controller needs a defined version, RBAC scope, metrics, logs, upgrade path, and behavior when dependencies are unavailable.
The scheduler chooses a node
The scheduler filters and scores nodes using resource requests, affinity, taints and tolerations, topology, volume constraints, policy, and installed scheduling extensions. It records a binding; it does not start the container.
A Pending Pod can mean no node satisfies all constraints, the workload has not been admitted by an external queue, or storage is waiting for placement. Inspect Pod events and the responsible scheduler or queue state before adding capacity.
Polyaxon scheduling presets and node scheduling make environment-specific placement rules reusable.
The node turns intent into processes
On the selected node:
- the kubelet watches assigned Pod specifications and drives their lifecycle;
- the CRI container runtime pulls images, creates sandboxes, and runs containers;
- network components configure Pod and Service traffic;
- CSI components attach and mount storage;
- device plugins or resource drivers expose accelerators;
- DaemonSets may provide logging, security, monitoring, and GPU management.
The kubelet does not choose the node; it reconciles Pods already assigned there. A FailedCreatePodSandBox event points toward the runtime, network, or node layer, while an image-pull error precedes application execution.
Add-ons complete the platform
Most usable clusters depend on add-ons: DNS, ingress or Gateway implementations, metrics collection, log shipping, policy engines, certificate management, storage drivers, autoscaling, and GPU components.
These are not “extra” when workloads depend on them. Inventory owners, versions, privileges, resource requests, and failure behavior. A cluster can report a healthy API server while a missing DNS or device-plugin Pod blocks useful ML work.
Debug by ownership
Use the workload's last successful transition to choose the next evidence:
| Symptom | Likely boundary to inspect |
|---|---|
| Object rejected immediately | Authentication, authorization, admission, API schema |
| Pod exists but remains unscheduled | Queue, scheduler, resources, affinity, taints, volume topology |
| Pod assigned but sandbox fails | Kubelet, runtime, CNI, node health |
| Container waits for image | Registry access, credentials, image, runtime, disk |
| Process starts then fails | Application command, dependencies, resources, configuration |
| Service has no endpoints | Selector and Pod readiness |
| Run completes but output is missing | Application persistence, artifact connection, cleanup lifecycle |
Capture context, namespace, object name, timestamps, owner, image digest, node, and recent change before modifying anything. A retry can erase the state that distinguishes a platform failure from application code.
Connect Kubernetes and Polyaxon
Kubernetes provides desired state, placement, and container lifecycle. Polyaxon adds the ML operation, project, inputs, parameters, tracking, artifacts, queues, and reusable execution policy.
Use Polyaxon platform observability alongside cluster telemetry. The combined model should let an operator trace one run from submission through API admission, scheduling, node execution, logs, metrics, termination, and persisted output.
Once component ownership is clear, Kubernetes becomes less mysterious. Each status transition is evidence from a particular control loop, and each failure has a smaller set of systems that could have produced it.