Troubleshoot FailedCreatePodSandBox errors

FailedCreatePodSandBox means the kubelet could not prepare the environment in which a Pod's containers should run. The failure happens before application startup, so changing the image command, readiness probe, or Python code will not fix it.

Most cases belong to one of four layers: the node's container runtime, the Container Network Interface (CNI) implementation, address allocation, or underlying node health. The event's detailed message and the distribution of failures tell you where to look.

Confirm the exact failure

Start with the affected Pod and keep the cluster context and namespace explicit:

kubectl --context production --namespace ml-team get pod training-run-abc123 -o wide
kubectl --context production --namespace ml-team describe pod training-run-abc123

Read the Events section from oldest to newest. Capture the complete message after FailedCreatePodSandBox; phrases such as “failed to setup network,” “failed to assign an IP,” “plugin not found,” or a runtime RPC error are more useful than the high-level reason.

Then establish the scope:

kubectl --context production get pods -A -o wide \
  --field-selector=status.phase=Pending
kubectl --context production get events -A \
  --sort-by=.lastTimestamp

If failures follow one node, suspect node-local runtime, CNI, filesystem, or configuration state. If new Pods fail across a node pool or cluster, investigate shared IP address management, a CNI rollout, API connectivity, or infrastructure changes.

Check node health before networking

Inspect the selected node:

kubectl --context production describe node worker-gpu-3
kubectl --context production get node worker-gpu-3 \
  -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}{"\n"}'

Look for Ready, NetworkUnavailable, DiskPressure, MemoryPressure, recent reboots, and taints. A full filesystem or unhealthy runtime can prevent sandbox creation even when the event mentions networking.

Use your managed Kubernetes provider's node diagnostics or the node team's approved tooling to inspect kubelet and runtime logs. The Kubernetes guide to debugging nodes with crictl is useful for operators with authorized node access. Do not introduce node-level access or privileged debugging Pods solely for convenience.

Inspect the CNI deployment

Kubernetes relies on a network plugin to configure Pod networking. The network plugin documentation describes the CNI integration; deployment names and components vary by provider and implementation.

Inspect system Pods and DaemonSets without assuming a particular vendor label:

kubectl --context production --namespace kube-system get daemonsets
kubectl --context production --namespace kube-system get pods -o wide

Find the network component expected on the affected node. Compare its readiness, restarts, image, and rollout revision with a healthy node. Check its logs through the provider-supported path. A running CNI Pod is necessary but not sufficient: configuration, host binaries, routes, or address allocation can still be wrong.

Avoid “testing” production by editing the CNI DaemonSet or moving files under /etc/cni. Those actions manufacture a wider outage and destroy useful evidence.

Check IP address allocation

Some failures occur because the plugin cannot allocate another Pod address. Depending on the environment, the exhausted resource could be a node-local pool, subnet, cloud-network interface, or provider-specific allocation object.

Compare:

scheduled Pods per affected node;
available subnet or pool addresses;
CNI/IPAM warnings and allocation errors;
recent node-pool growth or subnet changes;
leaked allocations after interrupted node lifecycle events.

Do not expand a subnet or delete allocation state until the responsible network component and ownership are clear. The repair must match the CNI and cloud-provider model.

Distinguish runtime from network failure

The kubelet asks the container runtime to create a Pod sandbox, and the runtime coordinates network setup. A runtime that is stopped, incompatible, or unable to write state may surface a similar high-level event.

Check the node's reported runtime version, service health, storage, and logs. If a recent node image or Kubernetes upgrade changed the runtime, compare a healthy and unhealthy node from the same pool. Our guide to Kubernetes container runtimes explains the CRI boundary in more detail.

Recover at the right layer

Once the cause is known, choose the narrowest supported recovery:

Cause	Appropriate owner and response
CNI Pod missing or unhealthy	Network/platform team restores the supported CNI deployment
Address pool exhausted	Network owner expands or reclaims capacity using provider procedures
One node has stale state	Node team cordons, preserves evidence, and replaces or repairs the node
Runtime unhealthy	Node team restores the configured CRI runtime or replaces the node
Disk or inode pressure	Reclaim space safely and correct the workload or node policy
Cluster-wide rollout regression	Stop the rollout and restore the last validated configuration

After repair, create a small representative Pod on the affected capacity, then verify the original workload. For GPU nodes, include a workload that exercises the device path; a generic web server proves only basic sandbox creation.

Polyaxon records the run, selected environment, status, and logs around ML operations. Correlate that context with node and CNI evidence so an infrastructure failure is not misclassified as a training-code failure. The durable fix is a monitored platform condition and a tested recovery procedure—not a manual retry that happens to work once.