Debug Kubernetes ML workloads
Debug ML workloads by following Kubernetes admission, scheduling, Pod startup, containers, nodes, networking, storage, and application outcomes in order.
Kubernetes debugging is faster when you follow the workload's lifecycle in order. A container log cannot explain an object rejected by admission, and adding nodes cannot fix a missing Secret or an invalid image reference.
For ML workloads, preserve the connection between the Kubernetes execution attempt and the logical run. Pods are replaceable; the code, data, parameters, resources, logs, metrics, and artifacts explain what the team intended to execute.
Capture context before changing anything
Record the cluster context, namespace, workload name, owner, timestamps, image digest, requested resources, selected node, and recent changes. Avoid a retry or manual edit until you have captured transient events and status.
Confirm the context explicitly:
kubectl config current-context
kubectl --context ml-production --namespace team-a get job training-run-42 --output=wideIf the object does not exist, check the platform submission, authentication, authorization, admission response, namespace, and API schema. A missing Pod may be correct when a queue has not admitted the workload or a controller has not created its dependents.
The official Kubernetes application troubleshooting guide organizes current debugging tasks without relying on deprecated component-status commands.
Follow ownership and conditions
Start with the highest-level object that represents the work, then follow owner references downward. A Job owns Pods; a Deployment owns ReplicaSets, which own Pods; a custom controller may own several workload objects.
kubectl --context ml-production --namespace team-a describe job training-run-42
kubectl --context ml-production --namespace team-a get pods --selector=polyaxon-run=training-run-42 --output=wideRead status conditions and events together. Conditions summarize state; events record recent decisions and failures. Events have limited retention, so capture them during the incident.
Changing a generated Pod is usually temporary. Fix the owning object or the Polyaxon configuration that produced it.
Diagnose Pending Pods before adding capacity
A Pending Pod can be waiting for several different reasons:
- insufficient CPU, memory, ephemeral storage, or accelerators;
- node selectors or affinity match no eligible node;
- taints are not tolerated;
- a PVC is unbound or storage topology conflicts with placement;
- quota, queue, or admission policy blocks progress;
- the selected scheduler or scheduling extension has not acted;
- autoscaling cannot provision a compatible node.
Use the scheduling event message as the starting hypothesis. Compare resource requests with allocatable capacity and inspect all constraints together. A cluster can report free GPUs while none have the required model, memory, topology, or storage access.
Polyaxon queues and node scheduling make admission and placement policy explicit.
Separate startup stages
Once a node is selected, several stages can fail before application code runs:
| Symptom | Boundary to inspect |
|---|---|
FailedCreatePodSandBox | Container runtime, CNI, node, or sandbox configuration |
ErrImagePull or ImagePullBackOff | Image name, registry, credentials, network, disk |
CreateContainerConfigError | Secret, ConfigMap, environment, or volume configuration |
| Init container failing | Initialization logs, command, dependency, or mounted data |
| Container repeatedly restarting | Exit reason, previous logs, probes, memory, application |
Retrieve current and previous logs separately:
kubectl --context ml-production --namespace team-a logs training-run-42-worker-0 --all-containers=true
kubectl --context ml-production --namespace team-a logs training-run-42-worker-0 --container=trainer --previousAn exit code is evidence, not a complete diagnosis. Combine it with the termination reason, signal, memory limits, node events, and application output.
Inspect nodes only when evidence points there
Node problems usually affect several workloads. Check readiness, pressure conditions, allocatable resources, kubelet and runtime health, disk, networking, time, and accelerator components.
If only one Pod fails on a healthy node, investigate its image, configuration, volumes, security context, and application before restarting shared services. If several unrelated Pods fail on the same node, cordon or drain according to the incident procedure after preserving evidence.
For GPU workloads, collect device-plugin status, driver version, runtime configuration, hardware health, memory errors, and topology. A visible device inside the container does not prove useful distributed-training connectivity.
Test network and storage from the right identity
DNS, Service selectors, NetworkPolicy, cloud firewalls, proxies, and egress can produce similar timeouts. Validate the request from a Pod with the same namespace, service account, and relevant policy as the failing workload.
For storage, separate provisioning, attachment, mounting, permissions, capacity, throughput, and application semantics. A bound PVC can still fail to attach in the selected zone or mount with an unexpected identity.
Do not use privileged debug containers as a default. Grant temporary access deliberately, log its use, and remove it when the investigation ends.
Preserve the ML-level explanation
Polyaxon run logging keeps output connected to the operation even when Kubernetes Pods are replaced. Artifact and run metadata show which code, parameters, inputs, and environment produced the attempt.
After recovery, record the failed boundary and add a prevention control: validation, admission policy, preset, alert, capacity rule, retry policy, or regression test. Platform observability should let an operator follow one run from submission through scheduling, execution, termination, and persisted output.
Debug in lifecycle order, preserve transient evidence, and change the true owner. That turns a vague “Kubernetes failure” into a specific boundary with a responsible system and a durable fix.