Troubleshoot OOMKilled in Kubernetes ML workloads

OOMKilled means the Linux out-of-memory mechanism terminated a process. In Kubernetes, the most common case is a container exceeding its memory cgroup limit, but node-wide memory pressure and application-level allocation behavior can produce related symptoms.

Increasing the limit may postpone the next failure without explaining the growth. Preserve the evidence, distinguish the scope, and measure a representative execution.

Confirm the termination state

Inspect the Pod and the specific container:

kubectl describe pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team

kubectl get pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --output jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'

Check init containers and sidecars as well as the main trainer. A log collector, data-preparation init container, or model server can be the process that exceeded its own limit.

Retrieve the previous container log before another restart replaces it:

kubectl logs training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --container trainer \
  --previous \
  --tail 300

An abrupt OOM kill may not let the application flush a final exception or checkpoint.

Distinguish requests from limits

A memory request is primarily a scheduling input. The scheduler uses requests to decide whether a Pod fits on a node. A memory limit is enforced by the kernel; when the container tries to use too much memory, the kernel can terminate a process.

If a limit is set without a request and no admission policy provides one, Kubernetes copies the limit as the request. The resource-management documentation describes these semantics.

Do not read “request” as a preallocated private block of memory. A process can use more than its request when node capacity is available, up to its effective constraints. Under node pressure, exceeding the request can affect eviction ranking.

Check container and node scope

A container-level OOM usually shows OOMKilled in its terminated state and usage near its memory limit. Node pressure can evict Pods or trigger system-level OOM behavior affecting several workloads.

Correlate:

• container working set, RSS, and configured limit;
• node available memory and MemoryPressure condition;
• Pod eviction events and node system logs;
• other workloads growing in the same time window;
• memory-backed emptyDir volumes, which count toward memory use;
• runtime and kernel metrics for OOM activity.

Current kubectl top output is not historical evidence:

kubectl top pod training-job-r8m4x \
  --context acme-production \
  --namespace ml-team \
  --containers

kubectl describe node gpu-node-17 \
  --context acme-production

Use the monitoring system to reconstruct the period before termination.

Look for ML-specific memory amplification

ML memory use is shaped by more than model size. Common causes include:

• data-loader worker processes duplicating datasets or caches;
• large prefetch queues and batch collation;
• decompression and parsing buffers;
• retaining tensors, predictions, or computation graphs;
• loading several model versions during a rollout;
• framework allocator caching and fragmentation;
• Python multiprocessing start methods;
• memory-mapped data becoming resident;
• an unbounded notebook or experiment result collection.

Measure the full process tree and container, not only the parent process. Reduce concurrency, batch size, prefetch depth, or retained state one variable at a time to identify the relationship.

GPU out-of-memory errors are separate: they occur in accelerator memory and are normally reported by the framework or driver. A process can fail from host RAM, GPU RAM, or both.

Choose a safer resource change

Set the request from the memory a workload normally needs, adjusted for scheduling policy and measured variance. Set the limit only after deciding how much burst is safe and what should happen when it is exceeded.

For batch training, a memory limit protects neighboring workloads but must include realistic peak phases such as dataset initialization and checkpoint serialization. For a replicated service, a tighter limit may be acceptable if one replica can restart without harming availability.

Avoid doubling resources blindly. A leak will consume the new headroom, while an inflated request can leave scarce GPU nodes underutilized because the scheduler cannot place other Pods.

Validate the recovery path

After changing code or resources, run a representative workload long enough to include the previous peak phase. Verify memory stabilizes, the operation makes progress, checkpoints remain valid, and no node-level pressure shifts the problem to other Pods.

For a controller-owned workload, change the operation or controller template. Editing a generated Pod does not update the source of truth.

Use Polyaxon for memory experiments

Polyaxon lets teams compare resource settings with execution outcomes. Record batch size, worker count, data strategy, image, node type, peak memory, duration, and failure reason for each operation. Promote validated settings into reusable components or scheduling presets.

The goal is not simply to make OOMKilled disappear. It is to establish a memory envelope that is repeatable, cost-aware, and compatible with reliable shared-cluster scheduling.