How to Fix: AWS ECS CannotPullContainerError

The Scenario

You’re deploying a new service via AWS ECS (Fargate or EC2 launch type) on a Friday afternoon. The CI/CD pipeline pushes a new image to ECR, updates the task definition, and triggers a service update. The new tasks fail immediately, rolling back the deployment. The only clue is a CannotPullContainerError in the stopped task logs.

Symptoms

• ECS tasks transition from PENDING to STOPPED within seconds.
• The task status reason in the AWS Console or CLI reads: CannotPullContainerError: <specific error message>.
• aws ecs describe-tasks --cluster <cluster> --tasks <task-id> shows stopCode: TaskFailedToStart and stoppedReason containing the pull error.
• No application logs are generated; the container never runs.

Root Cause

The ECS container agent on the host (or Fargate infrastructure) fails to download the specified container image. This is almost always due to one of three things: authentication failure (ECR or private registry), network failure (no route to the registry, DNS resolution failure, or a VPC endpoint issue), or image reference error (tag doesn’t exist, wrong repository URI).

Resolution (Step-by-Step)

1. Check the exact error message. The stoppedReason field is your best friend. Run:

   aws ecs describe-tasks --cluster <your-cluster> --tasks <task-id> --query 'tasks[0].stoppedReason' --output text

   This will give you the precise error, e.g., "CannotPullContainerError: Error response from daemon: pull access denied for <repo>" or "CannotPullContainerError: Head \"https://<account>.dkr.ecr.<region>.amazonaws.com/v2/<repo>/manifests/latest\": dial tcp: lookup <account>.dkr.ecr.<region>.amazonaws.com: no such host".

2. Fix ECR Authentication (most common cause). If the error mentions pull access denied or no basic auth credentials:

   • Ensure the task execution role (ecsTaskExecutionRole) has the AmazonECSTaskExecutionRolePolicy policy attached. This grants ecr:GetAuthorizationToken, ecr:BatchCheckLayerAvailability, ecr:GetDownloadUrlForLayer, and ecr:BatchGetImage.
   • Verify the role is correctly specified in the task definition under executionRoleArn.
   • If you’re using a private registry (Docker Hub, etc.), ensure the task definition has a repositoryCredentials block pointing to a valid AWS Secrets Manager secret containing the credentials.

3. Fix Network Access. If the error mentions a timeout, connection refused, or DNS resolution failure:

   • For Fargate: Ensure the task’s security group allows outbound HTTPS (port 443) to the internet (0.0.0.0/0) or to the VPC Gateway Endpoint for S3 (ECR uses S3 for layers). If using a NAT Gateway, verify the route tables are correct.
   • For EC2: Check the EC2 instance’s security group and NACLs for outbound HTTPS access. Also verify the instance can resolve DNS (e.g., nslookup <account>.dkr.ecr.<region>.amazonaws.com).
   • If you’re using VPC Endpoints for ECR (Interface Endpoints for ecr.api and ecr.dkr, and Gateway Endpoint for S3), confirm all three endpoints are in the same subnets as your tasks and that the security group attached to the Interface Endpoints allows inbound HTTPS from the task’s security group.

4. Verify the Image Reference. Ensure the image field in the task definition is an exact, existing tag or digest. A common mistake is using a tag that doesn’t exist (e.g., latest when the image was never pushed with that tag) or a typo in the URI.

   # Check if the image exists in ECR
   aws ecr describe-images --repository-name <repo> --image-ids imageTag=<tag>
   # Or for a digest
   aws ecr describe-images --repository-name <repo> --image-ids imageDigest=<sha256:...>

Why This Sometimes Doesn’t Work

The “fix the execution role” step is the most common solution, but it fails silently if the task role (not the execution role) is misconfigured. The execution role is for the ECS agent to pull images and send logs; the task role is for the application to make AWS API calls. If you accidentally changed the taskRoleArn instead of the executionRoleArn, the pull still fails. Double-check that the correct ARN is in the executionRoleArn field. Also, if you’re using a custom execution role, ensure it has a trust policy allowing ecs-tasks.amazonaws.com to assume it.

Verification

After applying the fix, force a new deployment of the service:

aws ecs update-service --cluster <cluster> --service <service> --force-new-deployment

Then watch the tasks:

aws ecs describe-services --cluster <cluster> --services <service> --query 'services[0].events' --output json

The events should show tasks moving to RUNNING without the CannotPullContainerError. You can also use:

aws ecs list-tasks --cluster <cluster> --service-name <service> --desired-status RUNNING

The count should match your desired count.

Common Follow-up Questions

Q: I’m getting CannotPullContainerError: AccessDenied but the role looks correct. What else could it be? A: Check if you’re using a cross-account ECR repository. The task execution role in your account must have a policy allowing ecr:GetDownloadUrlForLayer and ecr:BatchGetImage on the other account’s repository ARN. Also, the other account’s ECR repository policy must grant ecr:GetDownloadUrlForLayer to your execution role.

Q: The error says no basic auth credentials but I’m not using ECR. A: This usually means you’re pulling from Docker Hub or another public registry without specifying credentials. If the image is public, you don’t need credentials. If it’s private, you must configure repositoryCredentials in the task definition pointing to a Secrets Manager secret with the Docker Hub username and password in the format {"username":"...", "password":"..."}.

Q: The error is a timeout. I have a NAT Gateway. What gives? A: Timeouts can happen if the NAT Gateway is in a different availability zone than the task’s subnet, or if the NAT Gateway’s elastic IP is blocked by the registry. More commonly, it’s a DNS resolution issue inside the VPC. Ensure your VPC’s DHCP options set includes a valid DNS server (like the AWS provided one at 169.254.169.253).

Prevent This in the Future

The CannotPullContainerError is a classic deployment-time failure that can be caught before it hits production. Proactive monitoring of task state transitions and image pull latencies can alert you to underlying authentication or network degradation before a deploy fails. Teams use tools like Better Stack to track these metrics and get alerted on deployment failures in real-time, cutting down mean-time-to-resolution significantly. Set up uptime monitoring and alerts with Better Stack