How to Fix: GCP GKE ImagePullBackOff

The Scenario

You push a new container image to Google Container Registry (gcr.io) and deploy it to a GKE cluster. The pod stays in Pending for a few seconds, then flips to ImagePullBackOff. You see this on a Friday afternoon deploy, and your team’s on-call rotation just started.

Symptoms

kubectl get pods shows STATUS: ImagePullBackOff or ErrImagePull
kubectl describe pod <pod-name> shows events like Failed to pull image "<image>:<tag>": rpc error: code = Unknown desc = failed to pull and unpack image
Pod logs show Back-off pulling image or Error: ImagePullBackOff

Root Cause

The container runtime (containerd) on the GKE node cannot pull the specified image. This happens because: the image tag doesn’t exist in the registry, the registry requires authentication (e.g., private GCR), the node has no network access to the registry, or the image name/path is misspelled.

Resolution (Step-by-Step)

Check the exact image name and tag in your deployment manifest:
```
kubectl get deployment <deployment-name> -o yaml | grep -A2 "image:"
```
Verify the image path, tag, and registry domain (e.g., gcr.io, us.gcr.io). A common mistake: gcr.io/project-id/my-app:latest vs gcr.io/project-id/my-app (missing tag).

Test pulling the image manually on a GKE node (using a debug pod):

kubectl run debug-pod --image=busybox --restart=Never -- sleep 3600
kubectl exec -it debug-pod -- sh
# Inside the pod:
apk add --no-cache curl
curl -I https://gcr.io/v2/<project-id>/<image>/manifests/<tag>

If this fails, it’s a network or auth issue.

Verify registry authentication:
- For private GCR: ensure your service account has storage.objectViewer on the GCS bucket backing GCR.
- Check if you need an imagePullSecret:
```
kubectl get secrets
```
  If missing, create one:
```
kubectl create secret docker-registry gcr-json-key \
  --docker-server=gcr.io \
  --docker-username=_json_key \
  --docker-password="$(cat key.json)" \
  --docker-email=any@valid.email
```
  Then add imagePullSecrets to your pod spec.
Check GKE node’s network:
```
gcloud compute ssh <node-name> --zone <zone>
# On the node:
curl -I https://gcr.io/v2/
```
If blocked, check firewall rules or VPC Service Controls. Nodes in a private cluster need Cloud NAT for outbound internet.
Verify the image exists in the registry:
```
gcloud container images list-tags gcr.io/<project-id>/<image>
```
If the tag isn’t listed, rebuild and push the image.

Why This Sometimes Doesn’t Work

Even if you add the correct imagePullSecret, the secret must be in the same namespace as the pod. If you’re deploying to a non-default namespace like staging, but the secret is in default, the pull still fails silently. Also, GKE’s workload identity can override static secrets—check if your node pool uses workload identity and the associated service account has proper IAM permissions.

Verification

After applying the fix, watch the pod status:

kubectl get pods -w | grep <pod-name>

Expected output transitions from ImagePullBackOff → ContainerCreating → Running. Confirm with:

kubectl describe pod <pod-name> | grep -A5 "Events:"

Look for Successfully pulled image or no errors.

Common Follow-up Questions

Q: Why does it work on my local machine but not on GKE?
A: Your local Docker daemon may have cached credentials from gcloud auth configure-docker, but GKE nodes don’t. Always test with a fresh node or debug pod.

Q: How do I fix ImagePullBackOff for images in Artifact Registry?
A: Use the correct registry domain (e.g., us-central1-docker.pkg.dev) and ensure the node’s service account has artifactregistry.reader role.

Q: Can I force GKE to retry pulling the image?
A: Yes, delete the pod (kubectl delete pod <pod-name>) and let the deployment recreate it. Or scale the deployment to 0 and back.

Prevent This in the Future

Set up monitoring on pod status and image pull failures to catch these issues before they hit production. Better Stack provides real-time alerts on pod lifecycle events, letting you know the moment a pod enters ImagePullBackOff so you can fix registry or auth issues immediately. Set up uptime monitoring and alerts with Better Stack