Troubleshoot Kubernetes ImagePullBackOff errors

ImagePullBackOff means a kubelet could not pull a container image and is delaying subsequent attempts. It is a retry state, not a root cause.

The useful evidence is in the Pod specification and events: the exact image reference, which container failed, the registry response, the node, and the credential configuration available in that namespace.

Read the Pod and events first

Confirm the target and inspect the affected Pod:

kubectl get pod training-job-r8m4x \
  --context acme-staging \
  --namespace ml-team \
  --output wide

kubectl describe pod training-job-r8m4x \
  --context acme-staging \
  --namespace ml-team

The container may initially show ErrImagePull, then ImagePullBackOff as Kubernetes increases the retry delay. Read the event message rather than repeatedly deleting the Pod.

Identify all image references in a multi-container Pod:

kubectl get pod training-job-r8m4x \
  --context acme-staging \
  --namespace ml-team \
  --output jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.image}{"\n"}{end}'

Check init containers as well. An init-image failure prevents the application containers from starting.

Verify the repository, tag, or digest

Common reference failures include:

• a misspelled registry, repository, or tag;
• an image that was never pushed;
• a tag deleted by retention policy;
• a digest that does not exist in that repository;
• a registry hostname that resolves differently inside the node network;
• an image without a manifest for the node's operating system or architecture.

Confirm the digest produced by the build and promoted by the release process. Do not “fix” production by changing to latest; that removes the connection between the reviewed artifact and the running workload.

The Kubernetes container image documentation explains tags, digests, pull policy, and image names.

Check private-registry credentials

Private images require credentials available to the Pod through imagePullSecrets, the selected ServiceAccount, or a node-level credential provider.

Inspect names without printing secret contents:

kubectl get pod training-job-r8m4x \
  --context acme-staging \
  --namespace ml-team \
  --output jsonpath='{.spec.serviceAccountName}{"\n"}{.spec.imagePullSecrets[*].name}{"\n"}'

kubectl get serviceaccount workload-runner \
  --context acme-staging \
  --namespace ml-team \
  --output yaml

kubectl get secret registry-pull \
  --context acme-staging \
  --namespace ml-team

The Secret must exist in the same namespace as the Pod and its name must match exactly. The official private-registry guide covers this relationship.

An unauthorized or denied response may mean the credential is expired, scoped to another repository, missing a required registry audience, or blocked by organization policy.

Test the node-to-registry path

An image can pull successfully from a developer laptop while failing from a Kubernetes node. The kubelet and container runtime use the node's DNS, routes, proxy, certificate trust, firewall, and registry endpoint.

Use approved node or provider diagnostics to check:

• DNS resolution from the affected node pool;
• TCP and TLS connectivity to the registry;
• private endpoints, routes, and firewall rules;
• proxy and certificate-authority configuration;
• registry status and regional availability;
• disk space and inode availability in the image filesystem.

Compare affected and healthy nodes. If failures follow one subnet, zone, architecture, or node image, the Pod specification may be correct.

Do not place registry credentials in ad hoc debugging commands or terminal output.

Account for rate limits and concurrent pulls

Large ML images amplify registry and node pressure. A new GPU pool pulling the same multi-gigabyte image simultaneously can hit rate limits, network bottlenecks, unpacking limits, or disk pressure.

Read the event message for throttling or timeout evidence. Coordinate:

• authenticated registry limits;
• image-layer caching and node prewarming;
• rollout or autoscaling concurrency;
• image size and unnecessary build layers;
• regional mirrors where governance permits;
• node image-storage capacity and garbage collection.

Backoff is useful during a short registry outage. Recreating Pods in a loop increases load and discards the existing retry schedule.

Review pull policy and cached images

imagePullPolicy determines when the kubelet attempts a pull, while the image reference determines what it resolves.

Digest-pinned images make the intended artifact explicit. Nodes can reuse cached layers safely when their content matches. Mutable tags can produce different content across nodes and make a successful cached start look inconsistent with a failed fresh pull.

Choose pull policy and registry availability together. A policy that always resolves a remote tag makes every new Pod depend on registry reachability.

Check ownership and generated configuration

The failed Pod may be owned by a Job, Deployment, StatefulSet, operator, or Polyaxon operation. Fix the declaring layer, not only the generated Pod.

Inspect the owner references and compare the compiled image with the intended component:

kubectl get pod training-job-r8m4x \
  --context acme-staging \
  --namespace ml-team \
  --output jsonpath='{.metadata.ownerReferences[*].kind}{"\t"}{.metadata.ownerReferences[*].name}{"\n"}'

A live Pod edit cannot change an existing container image in place, and a controller would recreate the declared version anyway.

Prevent repeat failures with Polyaxon

Configure approved registries and credentials through Polyaxon connections and attach them only to eligible operations. Keep the image repository and digest in the versioned component or operation.

Track build provenance, scan result, registry, digest, source revision, and target architecture. Validate image existence and platform compatibility before an expensive queued workload reaches Kubernetes.

When an incident occurs, Polyaxon run metadata identifies the project, operation, agent, cluster, namespace, and image that failed. That context turns a generic Kubernetes status into an owned supply-chain problem.

Resolve ImagePullBackOff by fixing the underlying reference, authentication, network, registry, platform, or storage condition. Then verify that the controller creates a Pod using the intended immutable image and that the workload—not merely the pull—succeeds.