Troubleshooting articles

Control Kubernetes Job retries with pod failure policies

Stop retrying permanent errors, preserve the retry budget for marked disruptions, and inspect Kubernetes Job failure decisions with a concrete example.

Trace a failed training run to an unhealthy GPU

Connect Kubernetes DRA allocations, device health, and application failures without confusing missing telemetry with healthy hardware.

GPU jobs stuck Pending on Kubernetes: a debugging guide

Diagnose Pending GPU jobs by checking queue admission, scheduler events, advertised GPU resources, placement constraints, storage, and node capacity.

Troubleshoot Kubernetes CrashLoopBackOff

Use container state, previous logs, events, probes, configuration, and resource evidence to find the cause behind CrashLoopBackOff.

Troubleshoot OOMKilled in Kubernetes ML workloads

Diagnose container memory limits, node pressure, application allocation, and ML data-loading behavior before changing Kubernetes resources.

Restart Kubernetes Pods safely

Choose the correct restart path for Deployments, StatefulSets, Jobs, and standalone Pods while preserving evidence and workload ownership.