How to Fix: Kubernetes Node NotReady Taint

The Scenario

You’re running a production cluster (v1.28, 5-node pool) and your CI/CD pipeline triggers a rolling deploy on a Friday afternoon. After the third pod restarts, you notice deployments are stuck in Pending and kubectl get nodes shows one node with status NotReady. Your on-call phone buzzes — customers are hitting 503s because half your ingress replicas are stuck scheduling.

Symptoms

• kubectl get nodes shows one or more nodes as NotReady
• kubectl describe node <node-name> shows a taint: node.kubernetes.io/unreachable:NoSchedule or node.kubernetes.io/not-ready:NoSchedule
• Pods on the affected node show Evicted status, and new pods are stuck Pending with 0/1 nodes available: 1 node(s) had taint that the pod didn't tolerate

Root Cause

The Kubernetes control plane marks a node NotReady when the kubelet on that node fails to report its status (heartbeat) to the API server within the node-monitor-grace-period (default 40s). This triggers automatic tainting to prevent scheduling new pods onto the degraded node. Common underlying causes: the kubelet process has crashed, the node has run out of disk (eviction threshold), a network partition has cut off the node from the API server, or the container runtime (containerd/Docker) is hung.

Resolution (Step-by-Step)

1. Identify the node and get details:

   kubectl get nodes
   kubectl describe node <node-name> | grep -A5 Taints

2. SSH into the node and check kubelet status:

   ssh <user>@<node-ip>
   sudo systemctl status kubelet

   If kubelet is not running, start it:

   sudo systemctl restart kubelet
   sudo journalctl -u kubelet -n 50 --no-pager

3. Check disk pressure (common root cause):

   df -h
   # Look for /var/lib/kubelet or /var/lib/containerd at >85% usage
   # Clean up old container images or logs
   sudo crictl rmi --prune
   sudo journalctl --vacuum-time=2d

4. Verify network connectivity from the node to the control plane:

   curl -k https://<api-server-endpoint>:6443/healthz
   # Should return "ok"
   # If not, check firewall rules, VPN, or DNS resolution

5. If kubelet is healthy but node stays NotReady, manually remove the taint:

   kubectl taint nodes <node-name> node.kubernetes.io/unreachable:NoSchedule-
   kubectl taint nodes <node-name> node.kubernetes.io/not-ready:NoSchedule-

6. Force re-register the node by deleting and letting kubelet recreate it:

   kubectl delete node <node-name>
   # Kubelet on the node will re-register automatically within ~1 minute

Why This Sometimes Doesn’t Work

The manual taint removal or node delete trick often fails if the underlying cause is resource exhaustion (disk or memory) or a dead container runtime. You’ll clear the taint, the node will flip to Ready for 30 seconds, then immediately go back to NotReady. Always check journalctl -u kubelet -n 100 for eviction logs like Evicted Pod or NodeHasDiskPressure — if you see those, cleaning up disk space is the only fix that sticks. Another gotcha: if the node’s kernel is oom-killing processes, the kubelet will keep dying silently; check dmesg | tail -20 for OOM killer messages.

Verification

Run these commands in order:

kubectl get nodes <node-name>
# Expected: STATUS "Ready"
kubectl describe node <node-name> | grep Taints
# Expected: <none>
# Verify scheduling works:
kubectl run test-pod --image=nginx --restart=Never --overrides='{"apiVersion":"v1","spec":{"nodeName":"<node-name>"}}'
kubectl get pod test-pod
# Expected: STATUS "Running"
kubectl delete pod test-pod

Common Follow-up Questions

Q: Why does the node keep flipping between Ready and NotReady? A: Usually means the kubelet is crashing and restarting in a loop. Check journalctl -u kubelet -f for recurring error patterns, and look for resource pressure (disk, memory, PID limits) on the node.

Q: Can I prevent pods from being evicted from a NotReady node? A: Yes, set --pod-eviction-timeout on the kube-controller-manager to a higher value (default 5m). But this just delays the inevitable — fix the node health first.

Q: What’s the difference between unreachable and not-ready taints? A: unreachable means the control plane can’t contact the kubelet (network issue or kubelet crash). not-ready means the kubelet is reachable but reports unhealthy (e.g., disk pressure). Both block pod scheduling.

Prevent This in the Future

This class of issue is almost always avoidable with proactive monitoring of node health signals — kubelet uptime, disk usage, and API server connectivity. Setting up automated alerts for node status changes and resource pressure thresholds catches the problem before pods start being evicted. Teams commonly use Better Stack to monitor Kubernetes cluster health and get paged when nodes go NotReady, so they can intervene before production traffic is impacted. Set up uptime monitoring and alerts with Better Stack