Understand Kubernetes Pod evictions for ML

“Evicted” describes a Pod termination, but not every disruption follows the same mechanism or policy. Node pressure, a maintenance drain, scheduler preemption, taints, node failure, and a direct deletion have different causes and protections.

For ML workloads, the distinction determines whether Kubernetes retries one Pod, a distributed job must restart together, or a checkpoint can recover expensive progress.

Identify the disruption mechanism

Start from the Pod status and events:

kubectl get pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --output yaml

kubectl events \
  --context acme-production \
  --namespace ml-team \
  --for pod/training-job-r8m4x

Record .status.reason, .status.message, conditions, container termination states, the node, owner references, and event source. Events are short-lived and supplemental, so export them when investigations require history.

Do not assume every missing Pod was evicted. A controller rollout, completed Job cleanup, user deletion, or operator reconciliation may have removed it normally.

Understand node-pressure eviction

The kubelet monitors resources that cannot be recovered safely through ordinary CPU sharing: available memory, filesystem space and inodes, and process IDs. When configured thresholds are crossed, it reports node conditions such as MemoryPressure, DiskPressure, or PIDPressure and may reclaim resources and evict Pods.

Inspect the node without changing it:

kubectl describe node gpu-node-17 \
  --context acme-production

kubectl top node gpu-node-17 \
  --context acme-production

kubectl top is a recent resource view, not a reconstruction of the eviction moment. Correlate node metrics, container working sets, ephemeral-storage usage, filesystem inodes, and system logs. The Kubernetes node-pressure eviction documentation explains signals and thresholds.

Separate API eviction from direct deletion

Maintenance tools such as kubectl drain request eviction through the API. API-initiated eviction respects Pod disruption budgets (PDBs) and termination grace periods.

A direct Pod deletion is different. Node-pressure eviction is also different and is not prevented by a PDB. A PDB limits simultaneous voluntary disruption for selected replicated workloads; it does not guarantee that a Pod remains running.

Design PDB selectors carefully. A budget that selects no Pods protects nothing, while a budget that permits zero disruptions can block necessary maintenance.

Account for priority and preemption

When a high-priority Pod cannot schedule, the scheduler may preempt lower-priority Pods if removing them creates a feasible placement. Priority is an operational policy, not a statement that one experiment is universally more valuable.

Use a small, reviewed set of PriorityClasses. Reserve the highest classes for workloads with clear service or recovery objectives. If every team chooses the maximum, the mechanism cannot express useful tradeoffs.

Preemption may free CPU or memory but cannot invent an unavailable GPU type, storage zone, or required topology. Investigate the full scheduling constraint set.

Configure requests and local storage

Node-pressure decisions are affected by Pod priority, resource requests, and usage. Understated memory requests make placement denser and can leave bursty training processes competing on a node. Unbounded logs, writable layers, emptyDir, image storage, and checkpoints can trigger disk pressure.

Declare realistic requests for every container, including sidecars and init containers. Set ephemeral-storage requests and limits where appropriate. Send durable artifacts and checkpoints to persistent or object storage instead of relying on a Pod's writable layer.

Design ML workloads for interruption

A replacement Pod starts from declared state, not from the memory of the evicted process. Long-running training should write verified checkpoints at an interval derived from checkpoint cost, failure frequency, and recovery objective.

Distributed training needs coordinated failure behavior. If one worker disappears, the remaining processes may block while holding GPUs. Use a controller and framework strategy that terminates or recovers the group rather than waiting indefinitely.

Inference services need enough replicas, topology spread, readiness behavior, and graceful termination to preserve capacity during voluntary maintenance. Notebooks and interactive sessions need explicit expectations: disposable, resumable, or protected within a defined limit.

Prevent repeat evictions

Fix the condition, not the evidence. Depending on the cause, prevention may include:

• right-sizing requests and limits;
• reserving node capacity for system daemons;
• correcting eviction thresholds and filesystem layout;
• cleaning image and log growth safely;
• expanding or separating node pools;
• improving PDBs and maintenance procedures;
• adjusting workload priority and queue policy;
• checkpointing and retrying at the application layer.

Deleting the Evicted Pod object only cleans the API record. It does not relieve pressure or make the next attempt safer.

Coordinate recovery with Polyaxon

Polyaxon tracks the operation above individual Pods. Use retry policy, scheduling presets, resources, queues, and artifact connections to make recovery intentional. Preserve the failed run's node, reason, time window, and last checkpoint, then link any retry to that evidence.

An eviction is manageable when the platform can explain why capacity was reclaimed, how much work was lost, and whether the next operation will behave differently.