Restart containers selectively with Kubernetes restart rules

A worker loses a temporary connection and exits. Retrying that process may be reasonable. Retrying after an invalid configuration can produce an endless loop with no chance of success. A single restart policy cannot always express the distinction the application already knows.

Container restart rules let an application container select restart behavior by exit code. They are beta since Kubernetes 1.35 and enabled by default with ContainerRestartRules. Rules are checked in order; if none matches, the container's own restart policy applies. Pod lifecycle documentation.

This is recovery inside an existing Pod. Job failure policies instead govern how the Job controller responds to failed Pods. Choose the layer according to what must be recovered.

Define the application's exit contract

Start with a short table owned by the application team. For example:

These values are an example convention, not Kubernetes error classifications. A wrapper must preserve the program's intended exit code. If every exception becomes exit code 1, the scheduler cannot infer the underlying cause from the rule.

Also define what repeated execution means. Restarting a process can repeat an external write or partially completed action. Idempotent outputs and durable progress still belong to the application.

Observe one local restart

The fixture below requires Kubernetes 1.35 or later with ContainerRestartRules enabled, Pod creation permissions, and access to the Python image. Its worker records a marker in an emptyDir, exits 75 on its first attempt, and succeeds on the second. A separate observer container sleeps without participating in that retry.

Save it as container-restart-demo.yaml:

apiVersion: v1
kind: Namespace
metadata:
  name: container-restart-demo
---
apiVersion: v1
kind: Pod
metadata:
  name: selective-restart
  namespace: container-restart-demo
spec:
  restartPolicy: Never
  activeDeadlineSeconds: 180
  containers:
    - name: worker
      image: python:3.12-slim
      restartPolicy: Never
      restartPolicyRules:
        - action: Restart
          exitCodes:
            operator: In
            values: [75]
      command:
        - python
        - -c
        - |
          from pathlib import Path
          import sys

          marker = Path('/attempt/started')
          if not marker.exists():
              marker.write_text('first attempt reached the transient failure')
              print('Simulated temporary failure', flush=True)
              sys.exit(75)
          print('Second attempt completed', flush=True)
      volumeMounts:
        - name: attempt
          mountPath: /attempt
      resources:
        requests: {cpu: "100m", memory: "64Mi"}
        limits: {cpu: "500m", memory: "128Mi"}
    - name: observer
      image: python:3.12-slim
      command: ["python", "-c", "import time; time.sleep(60)"]
      resources:
        requests: {cpu: "100m", memory: "64Mi"}
        limits: {cpu: "500m", memory: "128Mi"}
  volumes:
    - name: attempt
      emptyDir: {}

The image tag is for a disposable example. Use approved digests in a maintained environment. The marker is deliberately Pod-local: emptyDir storage survives a container crash but is lost when the Pod is removed.

kubectl apply -f container-restart-demo.yaml
kubectl get pod selective-restart -n container-restart-demo -o yaml
kubectl logs selective-restart -n container-restart-demo -c worker
kubectl logs selective-restart -n container-restart-demo -c worker --previous

Inspect the logs after the retry has occurred. Under normal execution, the worker's restart count should reach one and the observer's remain zero. The Pod UID stays the same. The Pod is not fully complete until the observer has also exited. These are expected observations, not captured test results.

Decide when local recovery is insufficient

The restart reuses the Pod's node and allocation. If the node or assigned device is unhealthy, repeating the process there may repeat the failure. If multiple training workers must reset together, restarting one container can leave their distributed state inconsistent.

A restart rule also does not define a maximum attempt count. The fixture's marker makes it converge, while its Pod deadline provides an independent time bound. A real application needs a bounded recovery policy appropriate to its operation; an indefinitely recurring matching exit code can keep requesting restarts.

Use Job retries and completion accounting when the recovery unit is a failed Pod or a complete shard. Native sidecars have their own lifecycle conventions, described in the sidecar guide; do not treat every helper as an ordinary application container with interchangeable rules.

Remove the learning fixture afterward:

kubectl delete namespace container-restart-demo

Choose the recovery level with Polyaxon

Polyaxon provides resume and restart workflows when recovery needs to happen at the operation level. Resume makes the previous run's artifacts available again; restart can create a new run while preserving the original record, and copy mode carries the previous artifacts forward. You can also supply a preset to change the environment for the next attempt. Training code remains responsible for loading its checkpoint.

That creates a useful choice for a training or evaluation recipe. A brief recoverable process error may suit a selective restart on the existing allocation. A failure requiring fresh placement, a changed environment, or a separate run record belongs in the operation recovery workflow. Keep local restart counts and application attempts with the Polyaxon run's logs and final outcome so a successful run does not hide repeated process failures.

Polyaxon's environment configuration documents the Pod restart policy. Native container-level rules additionally require schema and controller support for their separate Kubernetes fields. If operation retries and local restarts are both enabled, calculate their combined retry exposure and make output writes safe across attempts.