Troubleshoot Kubernetes disk pressure

A node under disk pressure can stop accepting Pods and evict workloads that were already running. For ML teams, the cause is often not a persistent dataset volume. Large container images, downloaded model weights, checkpoints in emptyDir, container logs, and failed-run residue all compete for local ephemeral storage.

The right response is to identify which filesystem and resource crossed a kubelet threshold before deleting anything.

Confirm the node condition

Start with the scheduler and kubelet evidence:

kubectl --context production get nodes
kubectl --context production describe node worker-gpu-3

Look for the DiskPressure condition, node taints, eviction-related events, and allocatable ephemeral storage. Then inspect recent events across namespaces:

kubectl --context production get events -A \
  --sort-by=.lastTimestamp

Do not assume every Pending Pod on that node is caused by disk. Scheduler events may point to GPU availability, affinity, quota, or another taint. Keep the diagnosis tied to timestamps and concrete conditions.

Understand the eviction signals

The kubelet monitors filesystem and inode availability. The exact layout depends on the node and container runtime, but current Kubernetes can distinguish resources such as nodefs, imagefs, and containerfs. The node-pressure eviction documentation defines the signals and how hard or soft thresholds trigger reclamation and eviction.

The distinction matters:

• nodefs commonly holds logs, writable container layers, and emptyDir data;
• imagefs, when separate, holds container images and layers;
• containerfs, where supported, can separate writable layers from images;
• inode exhaustion can trigger pressure even when byte capacity appears available.

Check the configuration and filesystem layout used by your managed service or node image. A cleanup procedure copied from another cluster may target the wrong storage or interfere with the runtime.

Find the workload's local storage

Kubernetes accounts several sources as local ephemeral storage: writable container layers, node-level logs, and local emptyDir volumes. ML workloads often add avoidable pressure by treating local space as durable storage.

Review the affected Pod:

kubectl --context production --namespace ml-team describe pod training-run-abc123
kubectl --context production --namespace ml-team get pod training-run-abc123 -o yaml

Look for:

• emptyDir volumes used for datasets, caches, or checkpoints;
• missing ephemeral-storage requests and limits;
• verbose logs or repeated crash output;
• large images or many image variants on the same node pool;
• artifacts written locally but never uploaded;
• init containers that duplicate downloaded data.

Polyaxon artifact connections provide durable destinations for outputs. Local disk should be a bounded cache or workspace, not the only copy of a valuable checkpoint.

Stabilize before cleaning

First reduce additional pressure. Pause or route new workloads away from the affected pool according to your operational procedure. Preserve evidence for any failed or evicted runs. If maintenance requires draining a node, respect disruption budgets and the behavior of non-replicated workloads.

Then use provider-supported or node-team procedures to inspect and reclaim space. Typical candidates include unused images, rotated logs, and abandoned runtime data, but deletion should follow ownership and retention rules. Avoid deploying an ad hoc privileged Pod with host filesystem access simply to run cleanup commands; it expands the incident's security and blast radius.

If a node remains inconsistent after safe reclamation, replacing it from a known-good immutable image is often more reliable than repeatedly repairing it in place.

Set requests and limits

Declare local storage needs where the workload uses it:

resources:
  requests:
    cpu: "2"
    memory: 8Gi
    ephemeral-storage: 20Gi
  limits:
    memory: 12Gi
    ephemeral-storage: 40Gi

Requests help the scheduler account for capacity. Limits constrain consumption and can cause eviction when exceeded, so set them from measured behavior with headroom. Confirm that the node filesystem layout allows kubelet accounting; the Kubernetes documentation describes layouts that are not tracked correctly.

For predictable large data, use an appropriate persistent or remote storage system. For disposable caches, set a size limit where supported and make the application able to rebuild them.

Prevent recurrence

A durable prevention plan spans workload, node, and platform controls:

• alert on filesystem bytes, inodes, and the DiskPressure condition;
• track local storage by namespace, workload type, and node pool;
• cap log volume and configure rotation;
• remove unused image variants through supported garbage collection;
• separate storage-heavy workloads from latency-sensitive services;
• bound caches and emptyDir use;
• upload checkpoints and outputs during the run, not only at normal completion;
• test eviction and recovery behavior.

Correlate node pressure with Polyaxon run metadata and termination state. The useful question is not only “which node filled up?” but “which workload behavior produced the growth, what did Kubernetes evict, and can the work resume safely?”