Sep 17, 2024

How to fix exit code 137 in Kubernetes

Exit code 137 usually means the container was killed for memory pressure. Learn how to identify OOM kills and prevent repeat failures.

Exit code 137 usually means the process was killed with SIGKILL. In Kubernetes, that often points to memory pressure or an OOM kill. The container did not exit politely; the runtime stopped it.

For ML workloads, this is common. Data loaders, feature transforms, model initialization, and large batch sizes can push memory over the limit. The fix is not always "add more RAM". You need to know whether the limit is wrong, the workload is wrong, or the node is under pressure.

What is exit code 137?

All processes emit an exit code when they terminate. Exit codes provide a mechanism for informing the user, operating system, and other applications why the process stopped. Each code is a number between 0 and 255. The meaning of codes below 125 is application-dependent, while higher values have special meanings.

A 137 code is issued when a process is terminated externally because of its memory consumption. The operating system's out of memory manager (OOM) intervenes to stop the program before it destabilizes the host.

When you start a foreground program in your shell, you can read the ? variable to inspect the process exit code:

$ demo-binary
$ echo $?
137

As this example returned 137, you know that demo-binary was stopped because it used too much memory. The same thing happens for container processes, too-when a memory limit is being approached, the process will be terminated, and a 137 code issued.

Pods running in Kubernetes will show a status of OOMKilled when they encounter a 137 exit code. Although this looks like any other Kubernetes status, it's caused by the operating system's OOM killer terminating the pod's process. You can check for pods that have used too much memory by running Kubectl's get pods command:

$ kubectl get pods

NAME	READY	STATUS	RESTARTS	AGE
demo-pod	0/1	OOMKilled	0	2m05s

Memory consumption problems can affect anyone, not just organizations using Kubernetes. You could run into similar issues with Amazon ECS, RedHat OpenShift, Nomad, CloudFoundry, and plain Docker deployments. Regardless of the platform, if a container fails with a 137 exit code, the root cause will be the same: there's not enough memory to keep it running.

For example, you can view a stopped Docker container's exit code by running

$ docker ps -a

CONTAINER ID	IMAGE	COMMAND	CREATED	STATUS
cdefb9ca658c	demo-org/demo-image:latest	"demo-binary"	2 days ago	Exited (137) 1 day ago

The exit code is shown in brackets under the STATUS column. The 137 value confirms this container stopped because of a memory problem.

Causes of container memory issues

Understanding the situations that lead to memory-related container terminations is the first step towards debugging exit code 137. Here are some of the most common issues that you might experience.

Memory limits

Kubernetes pods will be terminated when they try to use more memory than their configured limit allows. You might be able to resolve this situation by increasing the limit if your cluster has spare capacity available.

Memory leaks

Poorly optimized code can create memory leaks. A memory leak occurs when an application uses memory, but doesn't release it when the operation's complete. This causes the memory to gradually fill up, and will eventually consume all the available capacity.

Natural increases in load

Sometimes adding physical memory is the only way to solve a problem. Growing services that experience an increase in active users can reach a point where more memory is required to serve the increase in traffic.

Resource contention

If multiple containers are competing for memory, they can starve each other of resources. This can lead to a situation where one container is terminated because it's using too much memory, even if it's not the root cause of the problem.

Requesting more memory than your compute nodes can provide

Kubernetes pods configured with memory resource requests can use more memory than the cluster's nodeshave if limits aren't also used. A request allows consumption overages because it's only an indication of how much memory a pod will consume, and doesn't prevent the pod from consuming more memory if it's available.

Preventing pods and containers from causing memory issues

Debugging container memory issues in Kubernetes-or any other orchestrator-can seem complex, but using the right tools and techniques helps make it less stressful. Kubernetes assigns memory to pods based on the requests and limits they declare. Unless it resides in a namespace with a default memory limit, a pod that doesn't use these mechanisms can normally access limitless memory.

Setting memory limits

Pods without memory limits increase the chance of OOM kills and exit code 137 errors. These pods are able to use more memory than the node can provide, which poses a stability risk. When memory consumption gets close to the physical limit, the Linux kernel OOM killer intervenes to stop processes that are using too much memory.

Making sure each of your pods includes a memory limit is a good first step towards preventing OOM kill issues. Here's a sample pod manifest:

apiVersion: v1
kind: Pod
metadata:
  name: pod-with-memory-limit
spec:
  containers:
    - name: container-with-memory-limit
    image: nginx:latest
    resources:
      requests:
      memory: "256Mi"
      limits:
      memory: "512Mi"

The requests field indicates the pod wants 256 Mi of memory. Kubernetes will use this information to influence scheduling decisions, and will ensure that the pod is hosted by a node with at least 256 Mi of memory available. Requests help to reduce resource contention, ensuring your applications have the resources they need. It's important to note, though, that they don't prevent the pod from using more memory if it's available on the node.

This sample pod also includes a memory limit of 512 Mi. If memory consumption goes above 512 Mi, the pod becomes a candidate for termination. If there's too much memory pressure and Kubernetes needs to free up resources, the pod could be stopped. Setting limits on all of your pods helps prevent excessive memory consumption in one from affecting the others.

Investigating application problems

Once your pods have appropriate memory limits, you can start investigating why those limits are being reached. Start by analyzing traffic levels to identify anomalies as well as natural growth in your service. If memory use has grown in correlation with user activity, it could be time to scale your cluster with new nodes, or to add more memory to existing ones.

If your nodes have sufficient memory, you've set limits on all your pods, and service use has remained relatively steady, the problem is likely to be within your application. To figure out where, you need to look at the nature of your memory consumption issues: is usage suddenly spiking, or does it gradually increase over the course of the pod's lifetime?

A memory usage graph that shows large peaks can point to poorly optimized functions in your application. Specific parts of your codebase could be allocating a lot of memory to handle demanding user requests. You can usually work out the culprit by reviewing pod logs to determine which actions were taken around the time of the spike. It might be possible to refactor your code to use less memory, such as by explicitly freeing up variables and destroying objects after you've finished using them.

Memory graphs that show continual increases over time usually mean you've got a memory leak. These problems can be tricky to find, but reviewing application logs and running language-specific analysis tools can help you discover suspect code. Unchecked memory leaks will eventually fill all the available physical memory, forcing the OOM killer to stop processes so the capacity can be reclaimed.

Final thoughts

Exit code 137 points to a hard kill. Start with memory limits, node pressure, container logs, and workload behavior before blindly increasing resources. Sometimes the fix is a larger limit. Sometimes it is a smaller batch size or a less wasteful preprocessing step.

Polyaxon gives each run logs, status, resources, and metadata, which makes these failures easier to connect to the workload that caused them instead of chasing anonymous pods.