How to Fix: Kubernetes OOMKilled Exit Code 137

The Scenario

You’re running a Node.js data processing job on a GKE cluster. It’s been stable for weeks, but after a recent data volume increase, the pod consistently crashes about 30 seconds after starting. kubectl describe pod shows the container exited with code 137, and kubectl logs cuts off mid-line.

Symptoms

kubectl get pods shows the pod in CrashLoopBackOff state.
kubectl describe pod <pod-name> shows State: Terminated with Reason: OOMKilled and Exit Code: 137.
The container’s logs end abruptly without a clean shutdown message.

Root Cause

The container’s memory consumption exceeded the resources.limits.memory defined in the Pod spec. The Linux kernel’s Out-Of-Memory (OOM) killer terminated the container’s main process to free up system memory. Exit code 137 (128 + 9) indicates the process was killed by SIGKILL (signal 9).

Resolution (Step-by-Step)

Check current resource limits:

kubectl describe pod <pod-name> | grep -A 2 Limits

Increase the memory limit in your Deployment/StatefulSet manifest:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"  # Increase from 256Mi to 512Mi

Apply the change:
```
kubectl apply -f deployment.yaml
```

If the issue persists, profile memory usage:

# Enable memory profiling in your application and expose a metrics endpoint
kubectl port-forward pod/<pod-name> 8080:8080
curl http://localhost:8080/debug/pprof/heap > heap.out

For Node.js specifically, add --max-old-space-size:

spec:
  containers:
  - command: ["node", "--max-old-space-size=400", "app.js"]

Why This Sometimes Doesn’t Work

Simply raising the limit often just delays the crash. The real issue might be a memory leak—your app is allocating memory and never releasing it. Watch the container_memory_working_set_bytes metric over time. If it steadily climbs until hitting the limit, you have a leak, not just a sizing issue. In that case, fix the leak first, then set a reasonable limit.

Verification

# Watch memory usage of the new pod
kubectl top pod <new-pod-name> --containers
# Expected: memory usage stabilizes below the new limit, pod stays Running

Common Follow-up Questions

Q: What’s the difference between exit code 137 and 139?
A: 137 is SIGKILL (OOM or manual kill -9), 139 is SIGSEGV (segmentation fault, usually a code bug).

Q: Can I disable OOM killing?
A: No, it’s a kernel safety mechanism. You can set memory.oom_grace_period in CRI-O, but this only delays killing—the pod will still crash eventually.

Q: I set resources.requests but not limits—why am I getting OOMKilled?
A: Without limits, the pod can use all node memory. If the node runs out, the kernel kills the biggest memory user. Always set both requests and limits.

Prevent This in the Future

Set up memory usage alerts before your pods hit the limit. Use a tool like Better Stack to monitor kube_pod_container_resource_limits and alert when usage exceeds 80% of the limit for 5 minutes. This gives you time to investigate before the OOM killer acts. Set up uptime monitoring and alerts with Better Stack