Restart Kubernetes Pods safely

Kubernetes does not have a universal “restart this Pod in place” operation. A Pod is an API object with a lifecycle; controllers normally replace it from a template. The safe action depends on whether the Pod belongs to a Deployment, StatefulSet, DaemonSet, Job, operator, or no controller at all.

Before restarting anything, capture the evidence that explains why a replacement might help.

Diagnose before changing state

Inspect status, ownership, events, and current and previous container logs:

kubectl get pod model-api-7c9dfc8d7b-k2qsm \
  --context acme-production \
  --namespace ml-services \
  --output wide

kubectl describe pod model-api-7c9dfc8d7b-k2qsm \
  --context acme-production \
  --namespace ml-services

kubectl logs model-api-7c9dfc8d7b-k2qsm \
  --context acme-production \
  --namespace ml-services \
  --container model-server \
  --previous \
  --tail 200

--previous is useful only when the container has restarted and the prior log is still available. Record the image digest, restart count, last termination reason, exit code, node, and recent configuration changes.

A replacement will not fix an invalid image, missing Secret, failing probe, incompatible model, exhausted volume, or inadequate memory limit.

Find the declaring owner

Read the Pod's owner reference:

kubectl get pod model-api-7c9dfc8d7b-k2qsm \
  --context acme-production \
  --namespace ml-services \
  --output jsonpath='{range .metadata.ownerReferences[*]}{.kind}{"\t"}{.name}{"\t"}{.controller}{"\n"}{end}'

A ReplicaSet usually points to a Deployment one level above it. A generated Pod may also be owned by a Job created by an operator or Polyaxon operation. Change or restart the layer that owns the desired state.

Restart a controller rollout

For Deployments, StatefulSets, and DaemonSets, kubectl rollout restart updates the Pod template and lets the controller replace Pods according to its rollout strategy:

kubectl rollout restart deployment/model-api \
  --context acme-production \
  --namespace ml-services

kubectl rollout status deployment/model-api \
  --context acme-production \
  --namespace ml-services \
  --timeout 10m

The official kubectl rollout reference lists supported resources. Check replica capacity, readiness, disruption budgets, and downstream dependencies before initiating a production rollout.

A rollout restart reuses the existing template. If the configuration source changed but the template did not, confirm how the application consumes it. Environment variables from a Secret are resolved when a new container starts; mounted Secret volumes can update eventually without a restart, but the application must reread the file.

Delete a controller-owned Pod only when appropriate

Deleting one controller-owned Pod causes its controller to create a replacement. This is useful for isolating a single unhealthy replica, but it is still a disruptive mutation:

kubectl delete pod model-api-7c9dfc8d7b-k2qsm \
  --context acme-production \
  --namespace ml-services \
  --wait=true

Confirm the owner, available replicas, termination grace period, and disruption policy first. Direct deletion does not use the Eviction API and is not a substitute for a coordinated node drain.

Never delete several replicas with an imprecise selector. List the matched objects and verify their owners before any bulk action.

Treat StatefulSets and Jobs differently

StatefulSet Pods have stable identities and commonly attach persistent volumes. Restart one ordinal at a time when the application requires quorum, leader handoff, or ordered recovery. Confirm the volume detached cleanly before forcing any storage operation.

A Job is intended to run to completion. Deleting its Pod can trigger a retry depending on the Job specification and remaining backoff budget. It can also repeat non-idempotent work. Inspect the Job's completion mode, retry policy, checkpoints, and side effects before replacing anything.

For distributed training, one worker replacement may invalidate the whole gang. Let the owning ML controller or Polyaxon manage the coordinated retry.

Handle standalone Pods carefully

A Pod without a controller is not recreated automatically after deletion. First preserve its manifest and understand why it was created directly. In production, migrate repeatable services and tasks to an appropriate workload controller rather than using standalone Pods as durable processes.

Do not use kubectl delete --force as a routine restart mechanism. Force deletion removes the API object without waiting for confirmation that the process stopped; another instance can start while the original is still running on an unreachable node.

Restart at the Polyaxon operation level

A Polyaxon operation owns more context than its Kubernetes Pod: component version, parameters, connections, artifacts, distributed roles, retries, and lineage. For experiments, training jobs, notebooks, and services launched through Polyaxon, use the operation lifecycle so replacements remain visible and reproducible.

The best restart is a controlled re-execution after the cause has been identified and the declaration corrected. Preserve the failed attempt, link the new run to it, and verify progress—not merely a new Pod name.