Kubernetes OOMKilled: diagnose exit code 137

Published 27 September 2026

Kubernetes OOMKilled means the Linux kernel terminated a process under memory pressure. Exit code 137 means the process received SIGKILL, but that number alone does not prove an out-of-memory event. Confirm Reason: OOMKilled, compare the container's peak working set with its limit, and check node pressure before raising anything.

Read the evidence in the right order

Start with the previous container state. A restarting container may be running normally by the time an engineer opens the incident, while its last state still records the reason, exit code and timestamps.

kubectl describe pod api-7d9f6b8f9c-x2k4p

kubectl get pod api-7d9f6b8f9c-x2k4p \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

kubectl get events --field-selector involvedObject.name=api-7d9f6b8f9c-x2k4p \
  --sort-by=.lastTimestamp

Exit status conventions commonly encode a fatal signal as 128 plus the signal number, so SIGKILL (9) appears as 137. Kubernetes documentation shows the pair Reason: OOMKilled and Exit Code: 137 for a container that exceeded its memory limit. A deployment, operator or person can also send SIGKILL, so preserve the reason and events instead of treating 137 as the diagnosis.

Container limit, node pressure or eviction?

Evidence	Likely mechanism	Check next
`OOMKilled` in the container's last state	The kernel killed a process under memory pressure, often at the container cgroup limit.	The container limit, working-set peak, kernel or runtime log and memory-backed volumes
Pod phase `Failed` with reason `Evicted`	The kubelet evicted the Pod because the node crossed a pressure threshold.	Node conditions, eviction message, competing Pods and system reservations
Exit code 137 without `OOMKilled`	The process received SIGKILL, but the sender is not established.	Rollout history, termination grace period, operator actions and node logs
Node became NotReady or several Pods died together	Node-wide memory exhaustion or another node failure may have affected several cgroups.	Kubelet, kernel and provider events for the same minute

A container can exceed its memory request without exceeding its limit. The scheduler uses requests to place Pods. The kernel enforces memory limits reactively through cgroups, and may terminate a process when memory pressure exposes the overage. That makes request and limit two different controls, not a soft and hard version of the same number.

Find the peak that the dashboard missed

kubectl top is useful for current usage, but a process can rise past its limit and disappear between samples. Query the monitoring system at its finest retained interval, and align the peak with the termination time. Separate working set, resident memory and cache if the runtime exports them.

kubectl top pod api-7d9f6b8f9c-x2k4p --containers
kubectl get pod api-7d9f6b8f9c-x2k4p \
  -o jsonpath='{range .spec.containers[*]}{.name}{" request="}{.resources.requests.memory}{" limit="}{.resources.limits.memory}{"\n"}{end}'
kubectl describe node worker-03

Check the workload shape around the peak: request concurrency, batch size, cache warm-up, queue depth, worker count and recent releases. A memory-backed emptyDir also counts against container memory. Kubernetes warns that an unbounded memory-backed volume can consume the Pod's full memory limit, or node memory when no limit exists.

For short-lived containers, capture a heap or runtime profile before the next kill if the application supports it. Repeated restarts can make average memory look healthy because every kill resets the process before the leak or backlog becomes visible.

Choose the fix from the cause

Fix an unbounded cache, leak or input-dependent allocation when usage should not keep growing.
Reduce concurrency or batch size when the workload is valid but its simultaneous allocations exceed the limit.
Raise the limit when measured legitimate peaks do not fit, and confirm the node can support the new total.
Raise the request with the limit when the Pod needs that memory predictably; otherwise the scheduler can pack the node for an unrealistically small footprint.
Set a sizeLimit on memory-backed emptyDir volumes and include their use in the memory budget.
Use in-place vertical resize or replace Pods through the workload controller when the resource change has been reviewed.

Removing every memory limit avoids a container-limit kill but moves the failure boundary to the node. One runaway process can then compete with the kubelet and unrelated workloads. Treat that as a deliberate isolation choice, not a troubleshooting shortcut.

What Kubernetes memory QoS changes in 1.37

Kubernetes MemoryQoS graduated to beta in version 1.37 and its feature gate is enabled by default. On Linux nodes using cgroup v2, it can configure memory.high throttling and tiered protection through memory.min and memory.low. The defaults do not turn either behavior on: memoryThrottlingFactor is null and memoryReservationPolicy is None until an operator changes the kubelet configuration.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9
memoryReservationPolicy: TieredReservation

Throttling can slow allocation before a cgroup reaches its hard limit. Tiered reservation gives Guaranteed Pods memory.min protection and Burstable Pods memory.low. Neither repairs a leak or makes physical memory larger. The reservation policy is node-wide, and the protected charge includes page cache, so test it on nodes that resemble the production workload before a broad rollout.

Upgrading to 1.37 does not automatically stop OOMKilled containers. The gate is on, but the default kubelet configuration writes no memory.high, memory.min or memory.low values. The original request, limit and workload behavior still decide the failure.

Close the incident with a reproducible check

Record the last termination reason, time, restart count and node.
Graph container and node memory across the minutes before the kill.
Identify whether the limit, node pressure or an external SIGKILL ended the process.
Change code, concurrency, request, limit or node capacity according to that evidence.
Replay the same workload and confirm both the peak and restart count stay within the agreed boundary.
Alert on repeated OOM terminations and memory pressure, not only current usage.

If the affected workload also requests accelerators, memory failure is separate from Kubernetes GPU scheduling: a Pod can be placed on the right GPU and still have an undersized memory limit. If clients report broken connections during every restart, use the TCP reset guide to confirm whether the endpoint, proxy or a middlebox ended the flow.

Sources

Resource management for Pods and containers: requests, limits, OOM enforcement, memory-backed volumes and the documented OOMKilled example
Kubernetes 1.37: Memory QoS graduates to beta: published 14 September 2026, defaults, configuration and limitations
Pod Quality of Service classes: Guaranteed, Burstable and BestEffort behavior under node pressure
Linux cgroup v2 documentation: kernel definitions of memory.min, memory.low, memory.high and memory.max

Kubernetes 1.37 feature state and documentation were checked on 27 September 2026.