How to Fix: AWS EKS Nodes NotReady

The Scenario

It’s 3 PM on a Wednesday, and you just pushed a minor node group scaling change via Terraform. Five minutes later, your CI pipeline starts failing because pods are stuck in Pending. You check kubectl get nodes and see two of your five t3.large nodes in the NotReady state. Your team’s Slack channel is already lighting up.

Symptoms

• kubectl get nodes shows NotReady status for one or more nodes, often accompanied by a SchedulingDisabled taint.
• kubectl describe node <node-name> shows conditions like KubeletNotReady or NetworkUnavailable.
• aws ec2 describe-instances shows the underlying EC2 instance as running, so the issue is at the Kubernetes layer, not the hypervisor.

Root Cause

A node enters NotReady when the kubelet fails to report its status to the control plane for more than node-monitor-grace-period (default 40 seconds). Common causes in EKS: the kubelet can’t authenticate to the cluster (expired or misconfigured node instance profile), the CNI plugin (aws-node) fails to assign pod IPs (subnet exhaustion or security group rules), or the kubelet itself is deadlocked due to disk pressure or a hung container runtime.

Resolution (Step-by-Step)

1. SSH into the affected node.

   # Get the instance ID from the node name
   INSTANCE_ID=$(kubectl get node <node-name> -o jsonpath='{.spec.providerID}' | cut -d'/' -f5)
   
   # Use Systems Manager Session Manager (preferred) or SSH with a key
   aws ssm start-session --target $INSTANCE_ID

2. Check kubelet logs for authentication or runtime errors.

   sudo journalctl -u kubelet -n 100 --no-pager

   Look for lines like failed to get token (auth issue) or container runtime is down (containerd/docker issue).

3. Verify the node’s IAM instance profile is valid and attached.

   # On the node
   curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/

   If empty, the profile is missing. Re-attach the correct eks-node-role from the AWS console or CLI.

4. Restart the kubelet and containerd.

   sudo systemctl restart containerd
   sudo systemctl restart kubelet

   Wait 60 seconds, then run kubectl get nodes from your local machine.

5. If still NotReady, check for subnet exhaustion (common with large deployments).

   # On the node, check aws-node logs
   kubectl logs -n kube-system -l k8s-app=aws-node --tail=50

   Look for failed to assign IP address or no IP addresses available. If found, add more subnets to the node group or increase the /28 subnet size.

6. As a last resort, cordon and drain the node, then terminate it.

   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   # Terminate the EC2 instance through the console or CLI

   The auto-scaling group will replace it.

Why This Sometimes Doesn’t Work

If you restart kubelet and it comes back healthy but the node goes NotReady again within 5 minutes, you’re likely hitting a race condition with the aws-node DaemonSet. The CNI plugin tries to initialize before the VPC CNI controller is ready, causing a permanent IP assignment failure. The fix isn’t another restart—you need to delete the aws-node pod on that specific node: kubectl delete pod -n kube-system aws-node-<hash>. The DaemonSet will recreate it, and the retry logic usually succeeds.

Verification

kubectl get nodes

Expected output:

NAME                                          STATUS   ROLES    AGE   VERSION
ip-10-0-1-101.us-west-2.compute.internal     Ready    <none>   12d   v1.28.5-eks-5e0fdde
ip-10-0-1-102.us-west-2.compute.internal     Ready    <none>   12d   v1.28.5-eks-5e0fdde

All nodes should show Ready. Run kubectl get pods -o wide to confirm pods are scheduled on the previously NotReady nodes.

Common Follow-up Questions

Q: What’s the difference between NotReady and Ready,SchedulingDisabled?
A: NotReady means the kubelet isn’t reporting; SchedulingDisabled is a manual taint (usually from kubectl cordon) used during maintenance. A node can be both.

Q: Can a node be NotReady because of a control plane issue?
A: Rarely. EKS manages the control plane for you. The issue is almost always at the node level—kubelet, CNI, or IAM.

Q: How do I check disk pressure on the node?
A: SSH in and run df -h. The kubelet evicts pods when disk usage exceeds 85% on the root volume. Look for EvictionThresholdMet in kubectl describe node.

Prevent This in the Future

Nodes going NotReady is a classic symptom of silent failures in your cluster’s foundation—expired IAM roles, exhausted subnets, or a misconfigured CNI. Proactive monitoring on kubelet heartbeat status and VPC IP utilization can catch these before they take down your workloads. Teams using Better Stack to aggregate node-level metrics and set up instant alerts for NotReady conditions often resolve these issues before anyone notices a deployment delay.

Set up uptime monitoring and alerts with Better Stack