How to Fix: Kubernetes ImagePullBackOff

The Scenario

You’re rolling out a critical microservice update on a Friday afternoon. The deployment YAML looks clean, kubectl apply succeeded, but the new pods are stuck in ImagePullBackOff. Your CI/CD pipeline (GitLab CI, pushing to ECR) shows a green build, but production is broken.

Symptoms

• kubectl get pods shows the pod status as ImagePullBackOff.
• kubectl describe pod <pod-name> shows events like: Failed to pull image "<image>:<tag>": rpc error: code = Unknown desc = Error response from daemon: manifest for <image>:<tag> not found: manifest unknown.
• kubectl logs <pod-name> returns nothing (the container never started).

Root Cause

Kubernetes cannot pull the specified container image from the registry. The kubelet on the node tried to pull the image, failed, and backed off (exponential backoff). This happens because: the image tag doesn’t exist, the image repository is private and no credentials are configured, the image registry is unreachable, or the image name/path is typo’d.

Resolution (Step-by-Step)

1. Check the exact error message:

   kubectl describe pod <pod-name> | grep -A 10 "Events:"

   Look for the Failed to pull image line. This tells you why the pull failed.

2. Verify the image exists in the registry:

   # For Docker Hub
   docker pull <your-image>:<tag>
   
   # For ECR
   aws ecr describe-images --repository-name <repo-name> --image-ids imageTag=<tag>
   
   # For GCR
   gcloud container images describe gcr.io/<project>/<image>:<tag>

   If the pull fails locally, the tag is wrong or the image doesn’t exist.

3. Fix the image tag in your Deployment:

   kubectl edit deployment <deployment-name>

   Correct the spec.template.spec.containers[0].image field. For example, change myapp:latest to myapp:v1.2.3.

4. Configure image pull secrets (for private registries):

   # Create a secret
   kubectl create secret docker-registry regcred \
     --docker-server=<your-registry-server> \
     --docker-username=<your-name> \
     --docker-password=<your-pword> \
     --docker-email=<your-email>
   
   # Add it to your deployment
   kubectl set image pull-secret <deployment-name> regcred

5. Check network connectivity to the registry (from a node):

   # SSH into a worker node
   curl -v https://<your-registry-server>/v2/

   If this fails, you have a firewall or DNS issue. Check VPC endpoints, NAT gateways, or proxy settings.

Why This Sometimes Doesn’t Work

The most common gotcha is that the image pull secret is created in the default namespace, but your pod runs in a different namespace. Secrets are namespace-scoped. Also, if you’re using a service account with attached pull secrets, ensure the service account exists in the pod’s namespace and the imagePullSecrets field is set on the pod spec, not just the service account. Another silent killer: the image tag latest is often cached locally on the node, leading to a stale image — always use explicit, immutable tags.

Verification

kubectl get pods -w

Watch the pod transition from ImagePullBackOff to ContainerCreating to Running. Also run:

kubectl describe pod <pod-name> | grep "Successfully pulled image"

You should see a success event.

Common Follow-up Questions

Q: My image pull secret is correct, but it still fails. What now? A: Check if the secret’s credentials have expired (e.g., a temporary AWS ECR token). Regenerate and reapply the secret. Also verify the secret is properly base64-decoded: kubectl get secret regcred -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d.

Q: I see Back-off pulling image but no specific error. A: This usually means the kubelet is retrying but the error is transient (e.g., registry rate limiting). Wait for the backoff to reset (typically 5 minutes) or check the kubelet logs on the node: journalctl -u kubelet -f.

Q: The image is public, why can’t it pull? A: Public registries (like Docker Hub) have rate limits for anonymous pulls. If your cluster has many nodes pulling at once, you’ll hit the limit. Create a free Docker Hub account and use a pull secret with your credentials.

Prevent This in the Future

Proactively monitor image pull failures and registry health to catch these issues before they hit production. Set up alerts for ImagePullBackOff events in your cluster. Better Stack provides robust incident management and uptime monitoring that can alert your team the moment a deployment starts failing, helping you roll back or fix the issue before users notice. Set up uptime monitoring and alerts with Better Stack