How to Fix: Azure AKS Node NotReady

The Scenario

You’re running a production AKS cluster (1.27.x) with 10 Standard_D4s_v3 nodes. Around 2:30 PM, your on-call alert for PodStatusUnknown fires. kubectl get nodes shows two nodes as NotReady. A CI/CD pipeline deploying a new microservice version just finished, and your team is getting 503s from the ingress controller.

Symptoms

• kubectl get nodes shows status NotReady for specific nodes.
• kubectl describe node <node-name> shows Ready condition is False with KubeletNotReady or NodeStatusUnknown.
• kubectl get pods -o wide shows pods stuck in Pending or Unknown on those nodes.
• Azure Portal shows the node VM as “Running” but the AKS node status is “Not Ready”.

Root Cause

The AKS node’s kubelet has stopped reporting its status to the control plane. This is almost always caused by one of three things: the kubelet process is hung or crashed (often due to disk pressure on the OS disk), the node’s network plugin (usually Azure CNI or Cilium) has failed preventing pod networking, or a critical node-level addon (like kube-proxy or azure-ipam) is in a crash loop. The node-lifecycle-controller sets the NodeReady condition to False after the node-monitor-grace-period (default 40s) elapses without a heartbeat.

Resolution (Step-by-Step)

1. Identify the specific node and get its details:

   kubectl get nodes
   kubectl describe node <node-name> | grep -A 5 Conditions

2. SSH into the node using AKS SSH (requires aad-pod-identity or direct VMSS access):

   # Get the VMSS instance ID
   az vmss list-instances --resource-group <MC_rg> --name <aks-node-pool-name> --query "[?name=='<node-name>']" -o tsv
   # SSH (requires bastion or direct RDP)
   ssh <user>@<node-private-ip>

3. Check kubelet status:

   sudo systemctl status kubelet
   sudo journalctl -u kubelet --since "15 min ago" --no-pager | tail -100

   Look for errors like failed to get stats from dockershim, eviction manager, or failed to run kubelet.

4. Check disk pressure (most common cause):

   df -h
   # Check specifically /var/lib/kubelet and /var/log
   du -sh /var/log/*

   If disk is 85%+ full, the kubelet will evict pods and eventually stop heartbeating. Clean logs:

   sudo journalctl --vacuum-time=2d
   sudo find /var/log -name "*.log.*" -mtime +2 -delete

5. Check CNI and container runtime:

   # Check containerd/docker status
   sudo crictl ps
   sudo crictl pods
   # Check CNI logs (Azure CNI)
   sudo journalctl -u azure-cni --since "15 min ago" --no-pager

6. Force restart the kubelet:

   sudo systemctl daemon-reload
   sudo systemctl restart kubelet

   Wait 2-3 minutes and check from your local machine: kubectl get nodes.

7. If restart fails, cordon and drain from control plane (last resort):

   kubectl cordon <node-name>
   kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
   # In Azure Portal or CLI, reimage the VMSS instance
   az vmss reimage --resource-group <MC_rg> --name <aks-nodepool-name> --instance-ids <instance-id>

Why This Sometimes Doesn’t Work

If the node is under severe disk pressure (e.g., /dev/sda1 is 100% full with container logs or Docker overlay2 layers), simply restarting kubelet will fail because it can’t write its status file. The journalctl vacuum command also fails when the journal itself is corrupted. In this case, you must SSH in, delete the largest files manually (sudo rm -rf /var/log/azure/*.gz or prune unused container images with sudo crictl rmi --prune), and then restart kubelet. Also, if the issue is a broken CNI plugin version (e.g., a bad azure-vnet-telemetry update), no amount of restarting helps — you need to roll back the node pool’s node image version via az aks nodepool upgrade --node-image-only.

Verification

kubectl get nodes -o wide | grep <node-name>
# Expected output: Ready status
kubectl describe node <node-name> | grep -E "Ready|Conditions"
# Expected: Ready=True, DiskPressure=False, MemoryPressure=False, PIDPressure=False

Common Follow-up Questions

Q: How do I check if this is a network plugin issue vs. kubelet issue? A: SSH into the node. If sudo systemctl status kubelet shows active (running) but kubectl get nodes shows NotReady, it’s likely the CNI. Check kubectl get pods -n kube-system | grep azure-npm or azure-vnet for CrashLoopBackOff.

Q: Can I automatically recover a NotReady node without manual SSH? A: Yes. Use AKS’s auto-repair feature (az aks update --enable-node-os-upgrade-channel NodeImage) or set up a cluster autoscaler that will replace unhealthy nodes. For immediate action, use the az vmss reimage command from step 7.

Q: What’s the worst-case scenario if I can’t SSH? A: You can delete the VMSS instance directly from Azure Portal, and the autoscaler will replace it (if enabled). Otherwise, scale the node pool down to 0 and back up: az aks nodepool scale --cluster-name <cluster> --name <nodepool> --node-count 0 then --node-count <original>.

Prevent This in the Future

Disk pressure and kubelet failures are often silent until they cascade. Setting up proactive monitoring on node-level metrics like disk usage, kubelet heartbeat latency, and pod eviction rates would catch this before pods go Unknown. Teams using Better Stack can configure alerting on these AKS node metrics and get notified within seconds of a status change, enabling response before the 40-second grace period expires.

Set up uptime monitoring and alerts with Better Stack