How to Fix: Kubernetes Pod Pending Insufficient CPU

The Scenario

It’s 3 PM on a Thursday. You just triggered a rolling update via your CI/CD pipeline (ArgoCD v2.8) to deploy a new microservice. The pod lands in Pending state immediately, and your Slack alerting pings within 30 seconds. kubectl get pods shows the pod stuck there, and kubectl describe pod reveals the classic: 0/5 nodes are available: 5 Insufficient cpu.

Symptoms

kubectl get pods -n <namespace> shows the pod stuck in Pending state for minutes or hours.
kubectl describe pod <pod-name> shows Events with Warning FailedScheduling and reason Insufficient cpu.
kubectl top nodes shows most nodes with >90% CPU allocation, but actual usage might be lower.

Root Cause

Kubernetes schedules pods based on resource requests, not actual usage. When the sum of requests.cpu across all pods on a node exceeds the node’s allocatable CPU capacity, the node is considered “full” for scheduling purposes. The scheduler has no available node that can satisfy the pod’s CPU request, so the pod is stuck in Pending. This is a capacity planning or resource fragmentation issue—not necessarily a CPU exhaustion problem at the OS level.

Resolution (Step-by-Step)

Check pod events for the exact reason
```
kubectl describe pod <pod-name> -n <namespace>
```
Look for Events section. It will list which nodes were considered and why they were rejected.
Inspect node resource allocation
```
kubectl describe nodes | grep -A5 "Allocated resources"
```
Look for % of Allocatable for CPU. If any node is >100%, you have oversubscription.

Identify pods consuming the most CPU requests

kubectl top pods --all-namespaces --sort-by=cpu

This shows actual usage. Then check requests:

kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu

Temporarily scale down non-critical pods
```
kubectl scale deployment <non-critical-deploy> --replicas=0 -n <namespace>
```
Or delete a test pod that has high CPU requests but low actual usage.

Increase node pool size (if on cloud)
For example, on GKE:

gcloud container clusters resize <cluster-name> --node-pool <pool-name> --num-nodes=<new-size> --region=<region>

Or add a new node pool with larger instance types.

Reduce CPU requests in your deployment manifest
Edit the deployment:

resources:
  requests:
    cpu: 250m   # Reduce from 500m if actual usage is lower

Then re-apply:

kubectl apply -f deployment.yaml

Use cluster autoscaler
Ensure the cluster autoscaler is enabled and has headroom to add nodes. Check logs:
```
kubectl logs -n kube-system deployment/cluster-autoscaler
```

Why This Sometimes Doesn’t Work

If you scale up nodes but the pod still stays Pending, check for node taints and tolerations. A common scenario: you add new nodes, but they have a taint (e.g., node.kubernetes.io/unschedulable:NoSchedule) that your pod doesn’t tolerate. Also, if your pod has nodeSelector or affinity rules that match zero existing nodes, scaling won’t help—you need to fix the selector. Another hidden gotcha: resource quotas at the namespace level (kubectl describe quota -n <namespace>) can block the pod even if nodes have capacity.

Verification

kubectl get pods -n <namespace> -w

Wait for the pod to transition from Pending to Running. Then verify CPU usage is within expected bounds:

kubectl top pod <pod-name> -n <namespace>

Common Follow-up Questions

Q: Why does kubectl top nodes show low CPU usage but pods are pending?
A: Scheduling is based on requests, not actual usage. You can have nodes at 20% actual CPU but 150% allocated requests. Use kubectl describe nodes to see allocated resources.

Q: Can I force a pod onto a node?
A: You can, but it’s a bad practice. Use kubectl taint to remove scheduling restrictions temporarily, or use nodeName in your pod spec for debugging only.

Q: How do I know if it’s a cluster autoscaler issue?
A: Check the autoscaler logs for scale-up events. If it logs no node pool can accommodate the pod, your node pool may be at max size or the pod’s resource request is larger than any single node can provide.

Prevent This in the Future

Set up proactive monitoring on kubectl describe nodes allocated resource percentages and alert when CPU requests exceed 80% of node capacity. This catches fragmentation before pods fail to schedule. Teams often use Better Stack to aggregate these metrics and trigger alerts before a deploy fails.
Set up uptime monitoring and alerts with Better Stack