Fix: OOMKilled (exit code 137) in Kubernetes

A container restarts again and again, or the pod shows OOMKilled or CrashLoopBackOff in kubectl get pods. kubectl describe pod shows the previous instance’s end:

    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Ready:          False
    Restart Count:  5

Exit code 137 means the process was killed with signal 9 (128 + 9). Reason: OOMKilled tells you the kernel’s out-of-memory killer did it because the container’s memory use went over its limits.memory. Memory limits are enforced by the kernel when it detects memory pressure, so a container can briefly exceed its limit before it is killed.

Step 1: confirm it is the container limit

kubectl get pod api-0 -n shop -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl get pod api-0 -n shop -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources}{"\n"}{end}'

Two other situations look similar but are different:

• Eviction: the pod status is Failed with reason Evicted and a message about the node being low on memory. The node ran out of memory as a whole, and the kubelet evicted pods to recover, preferring pods whose usage exceeds their requests. Fix by setting requests that reflect real use, so the scheduler does not overpack nodes.
• Exit code 137 without OOMKilled: something else sent SIGKILL, typically the kubelet after a failed liveness probe or after the termination grace period. Check events for Liveness probe failed.

Step 2: measure real memory use

kubectl top pod api-0 -n shop --containers
kubectl logs api-0 -n shop --previous --tail=50

kubectl top needs the metrics-server add-on. Watch usage over a normal load cycle: if it climbs steadily until the kill, suspect a leak; if it jumps during specific requests or batch jobs, the limit is simply too low for peak load.

Fix 1: set a realistic limit and request

resources:
  requests:
    memory: 512Mi
  limits:
    memory: 768Mi

kubectl set resources deployment/api -n shop -c api --requests=memory=512Mi --limits=memory=768Mi
kubectl rollout status deployment/api -n shop

Memory units: Mi and Gi are powers of two (mebibytes, gibibytes); M and G are powers of ten. 512m with a lowercase m means 0.512 bytes and is a common typo. Check that the new request still fits on your nodes, or the pod will stay Pending instead.

Fix 2: make the runtime respect the limit

Many language runtimes size their heap from the machine, not the container, or need explicit settings:

• JVM: current versions are container-aware; size the heap as a share of the limit with -XX:MaxRAMPercentage and leave room for non-heap memory (threads, metaspace, buffers).
• Node.js: set --max-old-space-size (in MiB) below the container limit.
• Go: set the GOMEMLIMIT environment variable slightly below the limit so the garbage collector works harder before the kernel steps in.
• Worker processes (web servers, Python with multiple workers): total memory is per-worker use times worker count. Reduce workers or raise the limit.

Fix 3: find and fix the leak

If usage grows without bound, a higher limit only delays the next kill. Take a heap profile with the language’s own tools while usage is high, and look for caches without size limits, unclosed connections and growing in-memory queues.

Things to avoid

• Removing the memory limit entirely. The container can then consume the node’s memory, and during node pressure containers without limits are more likely to be killed anyway.
• Setting the request far below the limit for memory-heavy apps. The scheduler packs pods by requests, so nodes become overcommitted and evictions follow.
• Restarting the pod repeatedly as a “fix”. The restart count will keep growing until the cause is addressed.

Official documentation

• Resource Management for Pods and Containers
• Assign Memory Resources to Containers and Pods