How to fix exit code 137 in Kubernetes

Exit code 137 commonly indicates that a process was killed with SIGKILL. In Kubernetes, an out-of-memory (OOM) kill is one possible cause. A forced shutdown or an explicit kill can produce the same exit code, so the number alone does not establish why the process stopped.

For ML workloads, this is common. Data loaders, feature transforms, model initialization, and large batch sizes can push memory over the limit. The fix is not always "add more RAM". You need to know whether the limit is wrong, the workload is wrong, or the node is under pressure.

What is exit code 137?

The shell or container runtime reports a status when a process ends. By the common shell convention, a fatal signal produces a status of 128 + signal number: SIGKILL is signal 9, giving 137. Applications can also explicitly exit with that value. See the Bash exit-status reference.

The Linux OOM killer can send SIGKILL when a container reaches its memory limit or the node runs out of memory. SIGKILL cannot be caught, so the application cannot finish cleanup or flush its final logs.

After a foreground program exits, $? holds its shell status. This illustrative session shows the number, without identifying its cause:

$ demo-binary
$ echo $?
137

For a container, inspect its termination reason as well as its exit code. kubectl get pods may display OOMKilled, as in this illustrative output:

$ kubectl get pods

NAME	READY	STATUS	RESTARTS	AGE
demo-pod	0/1	OOMKilled	0	2m05s

That display is a summary, not the Pod's API phase. A restarted container may already appear Running or CrashLoopBackOff, with the termination recorded in lastState. Use the real namespace, Pod, and container names to collect the details:

kubectl describe pod demo-pod -n default
kubectl get pod demo-pod -n default -o yaml
kubectl logs demo-pod -n default -c app --previous --timestamps
kubectl get events -n default --field-selector involvedObject.name=demo-pod --sort-by=.metadata.creationTimestamp

In status.containerStatuses, inspect state and lastState, including the reason, exit code, signal if present, and timestamps. Also check initContainerStatuses if an init container failed. --previous reads a restarted container's previous instance; use ordinary logs for a terminated container that has not restarted. Events and previous logs may expire, so preserve them promptly.

The same distinction applies to Docker, OpenShift, ECS, and other container environments: the exit code describes termination, while runtime and host evidence explain the cause.

For example, a stopped Docker container can appear as follows:

$ docker ps -a

CONTAINER ID	IMAGE	COMMAND	CREATED	STATUS
cdefb9ca658c	demo-org/demo-image:latest	"demo-binary"	2 days ago	Exited (137) 1 day ago

The 137 value is shown in brackets under STATUS. Inspect the container's state and its OOMKilled field for additional evidence:

docker inspect cdefb9ca658c

Distinguish memory kills from other forced terminations

Evidence around the failure	Likely direction	Next step
`OOMKilled`, a memory spike, and a container limit	Container memory exhaustion	Compare peak use with the limit and profile the failing workload
Node memory pressure, evictions, or kernel OOM records	Node-wide contention	Inspect node capacity, competing workloads, and resource requests
A rollout, deletion, or probe-triggered restart followed by a shutdown timeout	Graceful termination did not finish	Inspect signal handling, shutdown hooks, and the grace period
An explicit kill in operator or runtime records	Administrative or external termination	Identify the action and its trigger

Kubernetes normally gives a stopping container time to shut down. Processes still running after the grace period can receive SIGKILL. Ensure the application receives and handles its stop signal, keep preStop work bounded, and size terminationGracePeriodSeconds for measured shutdown behavior. Increasing memory will not fix a blocked shutdown hook. The Pod lifecycle documentation explains that sequence.

Causes of container memory issues

Once the evidence points to memory exhaustion, use the following causes to narrow the investigation.

Memory limits

Container memory limits are enforced reactively by the kernel. An allocation beyond the container's limit can trigger an OOM kill even when the node still has free memory. A larger limit can help if the workload genuinely needs it and the node has capacity; also review the request and workload concurrency before changing it.

Memory leaks

Poorly optimized code can create memory leaks. A memory leak occurs when an application uses memory, but doesn't release it when the operation's complete. This causes the memory to gradually fill up, and will eventually consume all the available capacity.

Natural increases in load

Growing services may need more memory to handle increasing traffic. Before adding capacity, check whether queues, batch size, worker count, or concurrent requests can be bounded. For training jobs, data-loader prefetching and copying a dataset into each worker can raise host memory use sharply.

Resource contention

If multiple containers are competing for memory, they can starve each other of resources. This can lead to a situation where one container is terminated because it's using too much memory, even if it's not the root cause of the problem.

Requesting more memory than your compute nodes can provide

A memory request informs scheduling; it does not cap consumption. The scheduler compares requests with node allocatable capacity and already scheduled requests. A Pod whose request cannot fit on any eligible node normally remains Pending; that is a scheduling problem, not an exit code 137.

A running workload can use more than its request, subject to its limit and available memory. Understated requests can therefore pack too many memory-hungry workloads onto one node.

Preventing pods and containers from causing memory issues

Debugging container memory issues is easier when you separate scheduling, runtime limits, and application demand. A container without an effective memory limit can consume available node memory, but that capacity is still finite. Namespace policies or admission settings may also supply defaults, so inspect the admitted Pod rather than only the manifest you submitted.

Setting memory limits

Memory limits contain a workload's impact on its neighbors, but a limit below normal peak demand causes repeat OOM failures. Size requests and limits together using representative workload evidence, and reserve node capacity for system services. Here is a valid illustrative Pod manifest; these numbers are not a sizing recommendation for every application:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-memory-limit
spec:
  containers:
    - name: container-with-memory-limit
      image: nginx:1.30.4-alpine
      resources:
        requests:
          memory: "256Mi"
        limits:
          memory: "512Mi"

The container requests 256 MiB for scheduling and has a 512 MiB memory limit. The scheduler uses resource accounting, not a guarantee that 256 MiB is currently unused or physically reserved. The container can use more than its request, but an allocation beyond its limit can cause an OOM kill independently of node-wide pressure. See Kubernetes resource management for the distinction between requests and limits.

Investigating application problems

Once your pods have appropriate memory limits, you can start investigating why those limits are being reached. Start by analyzing traffic levels to identify anomalies as well as natural growth in your service. If memory use has grown in correlation with user activity, it could be time to scale your cluster with new nodes, or to add more memory to existing ones.

If your nodes have sufficient memory, you've set limits on all your pods, and service use has remained relatively steady, the problem is likely to be within your application. To figure out where, you need to look at the nature of your memory consumption issues: is usage suddenly spiking, or does it gradually increase over the course of the pod's lifetime?

A memory usage graph that shows large peaks can point to poorly optimized functions in your application. Specific parts of your codebase could be allocating a lot of memory to handle demanding user requests. You can usually work out the culprit by reviewing pod logs to determine which actions were taken around the time of the spike. It might be possible to refactor your code to use less memory, such as by explicitly freeing up variables and destroying objects after you've finished using them.

Continual growth can indicate a leak, an unbounded cache, a growing queue, or retained allocator memory. A graph alone cannot distinguish them. Compare memory after equivalent work cycles, inspect heap or allocation profiles, and check whether usage stabilizes after load subsides. Reducing retained data, bounding caches, streaming input, or reducing worker concurrency may solve the problem more effectively than repeatedly raising the limit.

For an ML run, record the input size, batch size, worker count, image version, and the stage that failed alongside resource telemetry. A brief allocation spike can occur between monitoring samples, so a chart below the limit does not rule out an OOM kill. Also distinguish host RAM exhaustion from GPU device-memory errors; they require different evidence and fixes.

Final thoughts

Exit code 137 commonly points to a hard kill. Establish the cause from termination details, events, logs, and node evidence before changing resources. For an OOM failure, the fix may be a larger limit, a smaller batch size, or a less wasteful preprocessing step. For a shutdown timeout, fix the termination path.

Polyaxon connects each run's logs, status, configuration, and metadata. Use run monitoring to compare resource behavior with the run's parameters, and scheduling presets to apply the resulting resource policy consistently across similar workloads.