How to Fix: GCP GKE CrashLoopBackOff

The Scenario

Your Friday deployment pipeline pushes a new image to a GKE cluster running Kubernetes 1.28. The rollout looks fine for 30 seconds, then kubectl get pods shows three of your six replicas in CrashLoopBackOff. You’re on-call, the Slack channel is pinging, and the monitoring dashboard is turning red.

Symptoms

kubectl get pods shows CrashLoopBackOff for one or more pods, with a restart count climbing rapidly (e.g., RESTARTS: 5).
kubectl describe pod <pod-name> shows Back-off restarting failed container in the Status section.
Container logs (kubectl logs <pod-name> --previous) show a process exit or a startup failure, often with an exit code like 1, 137, or 139.

Root Cause

The CrashLoopBackOff state means the container inside the pod starts, crashes, and is repeatedly restarted by the kubelet, with an exponential back-off delay. This is almost always caused by one of three things: a misconfigured liveness/readiness probe that kills the container prematurely, an application crash due to missing dependencies or config, or a resource limit (CPU/memory) so low the OOM killer terminates the process.

Resolution (Step-by-Step)

Check the exit code and reason.
```
kubectl get pods --namespace <namespace>
kubectl describe pod <pod-name> --namespace <namespace>
```
Look for Last State: Terminated and its Exit Code. Exit code 137 = OOM killed (SIGKILL). Exit code 139 = segmentation fault (SIGSEGV). Exit code 1 = generic app error.
Inspect the container logs from the last run.
```
kubectl logs <pod-name> --previous --namespace <namespace>
```
This fetches logs from the crashed instance before the restart. Look for stack traces, connection refused errors, or missing file messages.
Verify liveness and readiness probe configuration.
```
kubectl get pod <pod-name> -o yaml --namespace <namespace> | grep -A 10 livenessProbe
```
If the initialDelaySeconds is too short or the probe path/port is wrong, the pod gets killed before it finishes booting. Fix by adjusting initialDelaySeconds (e.g., 15-30) or the probe endpoint.
Check resource requests and limits.
```
kubectl describe pod <pod-name> --namespace <namespace> | grep -A 2 Limits
```
If memory limits are set too low (e.g., 128Mi for a Java app), the container gets OOM-killed. Increase the limit in your Deployment manifest, or remove the limit temporarily to test.
Manually run the container locally to isolate the issue.
```
docker run --rm <image>:<tag> /bin/sh -c "your-entrypoint-command"
```
This bypasses Kubernetes and confirms whether the image itself is broken (e.g., missing env vars, bad startup script).

Why This Sometimes Doesn’t Work

If the container crashes faster than kubectl logs --previous can capture output (e.g., it exits in under 100ms), you won’t see any logs. In that case, check the pod’s events with kubectl describe pod—the Events section often has a Started container followed by Back-off restarting without any error output. This usually points to an immediate segfault or a missing shared library. Run the image locally with docker run --entrypoint /bin/sh -it <image> to inspect the filesystem.

Verification

Confirm the fix by watching the pod status stabilize:

kubectl get pods --namespace <namespace> --watch

Expected output: the pod transitions from Pending → Running and the RESTARTS column stops incrementing. Run kubectl logs <pod-name> to see healthy application output.

Common Follow-up Questions

Q: How do I tell the difference between an OOM kill and a probe failure? A: Check the Last State exit code in kubectl describe pod. Exit code 137 (SIGKILL) means OOM. Exit code 1 or 143 means the app exited normally or was killed by a probe.

Q: What if the pod is in CrashLoopBackOff but has no logs at all? A: The container probably failed before any output was written. Run the image locally with docker run and override the entrypoint to /bin/sh to debug interactively.

Q: Can a ConfigMap or Secret change cause this? A: Yes. If your app reads config from a mounted ConfigMap or Secret that’s missing a key or has bad YAML, it will crash on startup. Compare the mounted volume contents with kubectl exec on a healthy pod.

Prevent This in the Future

Most CrashLoopBackOff events are silent until they hit production—you’re relying on someone noticing a dashboard or pager. Proactive monitoring that catches pod restarts and container exit codes before they cascade into a full outage is essential. Teams use tools like Better Stack to alert on abnormal restart counts and probe failures, so you can fix the root cause before it affects users. Set up uptime monitoring and alerts with Better Stack