Fix: Pod stuck in Pending with “0/3 nodes are available: Insufficient cpu”

The pod stays in Pending, has no node assigned, and kubectl describe pod shows a FailedScheduling event like this (numbers vary with your cluster):

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12s   default-scheduler  0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Read it as: of 3 nodes, 0 can take the pod; on all 3 the reason is not enough CPU. The part after preemption: says the scheduler also could not make room by evicting lower-priority pods. When different nodes fail for different reasons, they are listed together, for example 1 Insufficient cpu, 1 node(s) had untolerated taint(s); the same message with Insufficient memory means the same thing for RAM.

Why it happens: requests, not usage

The scheduler does not look at how busy a node actually is. It adds up the requests of all pods already on the node and checks whether the new pod’s request still fits into the node’s allocatable capacity (total capacity minus what is reserved for the system and kubelet). A node can be nearly idle in kubectl top and still be “full” for scheduling.

Step 1: compare the request with what nodes can offer

kubectl get pod web-0 -n shop -o jsonpath='{range .spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
kubectl describe node worker1.example.com | grep -A 8 "Allocated resources"

CPU is measured in cores: 500m is half a core, 2 is two cores. If a single container requests more than any node’s allocatable CPU (say cpu: 4 on 2-core nodes), the pod can never be scheduled, whatever else you do.

Fix 1: right-size the request (most common)

Requests are often copied from a template or a Helm chart’s defaults and are far higher than needed. Set them close to the real steady-state use, which you can measure with kubectl top pod --containers over time:

resources:
  requests:
    cpu: 250m
    memory: 256Mi
  limits:
    memory: 512Mi

Also check the namespace for a LimitRange that injects default requests into containers that do not set them: kubectl get limitrange -n shop -o yaml.

Fix 2: free capacity

• Find which pods reserve the most CPU on each node with the Allocated resources and Non-terminated Pods sections of kubectl describe node.
• Scale down or remove workloads that are no longer needed. Warning: check with their owners first; scaling down production services causes outages.
• Look for leftovers: finished test deployments, forgotten replicas, oversized sidecars injected into every pod.

Fix 3: add capacity

Add nodes, or larger nodes. If the cluster autoscaler is enabled, it reacts to exactly this kind of pending pod, but only up to the node group’s maximum size; its events on the pod say why it did not scale up.

Fix 4: use priority deliberately

A pod with a higher priorityClassName can preempt (evict) lower-priority pods to make room. “No preemption victims found” means there was nothing lower-priority to evict, or evicting it would not free enough or would break a PodDisruptionBudget. Give critical workloads a higher PriorityClass rather than raising everyone’s priority.

If CPU is not the only reason

• node(s) had untolerated taint(s): the nodes are tainted (for example control-plane nodes) and the pod lacks a matching toleration.
• node(s) didn't match Pod's node affinity/selector: check nodeSelector and affinity rules against kubectl get nodes --show-labels.
• Too many pods: the node has reached its maximum pod count.
• Messages about unbound PersistentVolumeClaims: the pod waits for storage, not CPU; check kubectl get pvc.

Official documentation

• Resource Management for Pods and Containers
• Assign CPU Resources to Containers and Pods
• Debug Pods