Troubleshoot Kubernetes CrashLoopBackOff

CrashLoopBackOff means a container has terminated repeatedly and the kubelet is delaying another restart. The backoff protects the node from an immediate restart loop. It does not identify why the process exited.

The cause is usually visible in the container's last termination state, previous logs, Pod events, declared command and configuration, probes, or resource history.

Identify the failing container

A Pod can contain init containers, application containers, and sidecars. Find the container with a waiting state or growing restart count:

kubectl get pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --output jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.state.waiting.reason}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'

kubectl describe pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team

Repeat the status check for .status.initContainerStatuses when initialization never completes. A healthy main container cannot start until required init containers succeed.

Record the image ID and digest, command, arguments, environment sources, volume mounts, probes, node, owner, and recent events.

Read the previous container log

The current container may be waiting or may have started with an empty log. Retrieve the log from the previous instance:

kubectl logs training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --container trainer \
  --previous \
  --timestamps \
  --tail 300

Capture it promptly because only limited previous-container history is available on the node. Check application telemetry and centralized logs for earlier context.

Exit code 1 is a generic application failure. 137 often means the process received SIGKILL, but confirm the Kubernetes termination reason and node evidence rather than assuming it was an OOM kill. 143 commonly reflects graceful SIGTERM handling.

Compare declared command and image

Container startup can fail because an executable is missing, the working directory changed, an argument is invalid, a script lacks permission, or the image targets the wrong architecture.

Compare the Pod's compiled command with the tested component and immutable image digest. If the image's entrypoint already supplies a command, understand how Kubernetes command and args override it.

For a distroless or immediately crashing image, the Kubernetes kubectl debug workflow can create a copy with a modified command. Use an approved debugging image and keep production credentials out of the copied Pod.

Validate configuration and dependencies

A container may start and exit because a Secret key, ConfigMap field, model artifact, dataset, DNS name, or external endpoint is unavailable. Check references without printing sensitive values:

kubectl get secret object-store-reader \
  --context acme-production \
  --namespace ml-team

kubectl get configmap trainer-config \
  --context acme-production \
  --namespace ml-team

Confirm the referenced objects exist in the same namespace, key names match, the ServiceAccount is correct, volume mounts are ready, and network policy permits required connections. Separate “dependency temporarily unavailable” from “invalid immutable configuration” when deciding whether a retry is useful.

Inspect startup and liveness probes

A liveness probe can terminate a process that is still initializing. ML services may need time to load a large model, compile kernels, warm caches, or establish accelerator state.

Use a startup probe to protect a slow but bounded initialization phase. Use readiness to control traffic and liveness only for conditions a process restart can repair. Review probe path, port, scheme, timeout, period, failure threshold, and the resource pressure under which the probe runs.

The event stream normally shows repeated probe failures before restarts. Do not “fix” the loop by removing health checks without replacing them with correct application health semantics.

Check memory and node conditions

If the last state is OOMKilled, compare memory usage with the container limit and investigate application allocation. If the node reports MemoryPressure, DiskPressure, or PIDPressure, the problem may affect several workloads.

kubectl top pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --containers

kubectl describe node gpu-node-17 \
  --context acme-production

These commands show current state. Use historical metrics for the period before the crash. Also check writable-layer and emptyDir growth, especially for verbose logs, downloaded datasets, temporary checkpoints, and model caches.

Fix the declaring layer

Find the Pod's owner:

kubectl get pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --output jsonpath='{range .metadata.ownerReferences[*]}{.kind}{"\t"}{.name}{"\n"}{end}'

Correct the Deployment, Job, StatefulSet, operator resource, or Polyaxon operation. Deleting the Pod only starts the same declaration again and can erase useful evidence.

Validate the replacement through the previous failure phase. A restart count of zero is not enough; the workload must become ready or make expected training progress.

Diagnose loops through Polyaxon

Polyaxon preserves the component, image, parameters, connections, resources, scheduling configuration, and operation history above the Pod. Compare the failed attempt with the last successful run and link the replacement to the change that addresses the cause.

Use retry policy for transient failures, not deterministic configuration errors. CrashLoopBackOff is resolved when the operation can start and continue correctly—not when the next backoff interval has merely begun.