Troubleshoot Kubernetes nodes in NotReady state

A Kubernetes node shown as NotReady is not reporting a healthy Ready condition to the control plane. New Pods normally stop scheduling there, and existing workloads may eventually be evicted or replaced according to node-controller behavior and their tolerations.

Do not begin by rebooting or deleting the node. Preserve evidence, identify whether the control plane can hear the kubelet, measure workload impact, and confirm who owns the machine before changing state.

Understand Ready, False, and Unknown

The official Kubernetes node documentation describes node status, conditions, heartbeats, and controller behavior.

The Ready condition can be:

• True: the node is healthy enough to accept Pods;
• False: the kubelet reports that the node is not healthy;
• Unknown: the control plane has stopped receiving the expected heartbeat.

kubectl get nodes summarizes both False and Unknown as NotReady. Read the condition reason, message, transition time, and related events before forming a hypothesis.

Pressure conditions such as MemoryPressure, DiskPressure, and PIDPressure are separate signals. Accelerator availability is separate again: a node can be Ready while its GPU device plugin is unhealthy or no GPU is allocatable.

Establish scope and timing

Start with cluster-scoped, read-only commands:

kubectl get nodes \
  --context acme-production \
  --output wide

kubectl describe node gpu-worker-07 \
  --context acme-production

Record when readiness changed, which conditions changed first, recent events, node version, container runtime, internal address, taints, capacity, and allocatable resources.

Ask whether one node, one pool, one zone, or the entire cluster is affected. A simultaneous failure points toward shared networking, control-plane reachability, identity, DNS, cloud, or automation rather than an isolated machine.

Check heartbeats and control-plane reachability

Nodes report status and renew Lease objects in the kube-node-lease namespace. Inspect the affected Lease:

kubectl get lease gpu-worker-07 \
  --context acme-production \
  --namespace kube-node-lease \
  --output yaml

A stale Lease supports a communication or kubelet hypothesis but does not identify the cause. Check the node-to-API-server path, routing, firewall rules, certificates, time synchronization, DNS where relevant, and cloud network health.

Do not assume internet access is required. The important path is between node components and the configured cluster services.

Inspect workload impact

List Pods assigned to the node across namespaces:

kubectl get pods \
  --context acme-production \
  --all-namespaces \
  --field-selector spec.nodeName=gpu-worker-07 \
  --output wide

Identify critical services, local persistent volumes, singleton workloads, distributed training groups, and Pods protected by disruption budgets. A training worker lost from one node can stall healthy peers while they continue reserving GPUs.

Check whether controllers created replacements and whether those replacements are Pending because another eligible node does not exist.

Investigate the kubelet and runtime

Node-level access depends on the provider and operating system. Use the approved administrative path to inspect:

• kubelet service state, logs, and certificate errors;
• container runtime state, disk use, and image filesystem;
• operating-system memory, PID, filesystem, and inode pressure;
• kernel, cgroup, mount, and time-sync errors;
• recent configuration, image, or node-agent changes.

The kubelet is responsible for node registration, heartbeat, and Pod reconciliation. A failed container runtime may prevent workloads from running even while some node communication continues.

Avoid clearing runtime state or deleting files until logs and disk evidence are preserved. Cleanup can remove the data needed to identify recurrence.

Check cluster networking and node agents

Inspect the CNI, DNS, storage, device-plugin, and node-observability DaemonSets relevant to the affected pool:

kubectl get daemonsets,pods \
  --context acme-production \
  --namespace kube-system \
  --output wide

Filter the Pod list to the node and read logs from the exact failing container. Not every cluster uses kube-proxy, and a failed node add-on does not always change node readiness directly. Treat each signal as evidence, not a universal cause.

Check whether admission or rollout automation deployed an incompatible node agent to the entire pool.

Review cloud and hardware health

For managed clusters, compare Kubernetes evidence with instance, autoscaling-group, maintenance, spot-interruption, disk, network-interface, and health-check events from the provider.

For physical nodes, inspect power, disks, memory, network interfaces, firmware, and accelerator health. GPU resets, driver faults, or fabric errors may require draining the workload even if the Kubernetes Ready condition remains True.

Distinguish a replaceable worker from a stateful or specialized machine. Node replacement is useful only when provisioning recreates the correct drivers, labels, taints, storage access, and observability.

Remediate with an owned recovery path

Choose the smallest action supported by evidence:

• restore API connectivity or credentials;
• recover the kubelet, runtime, filesystem, or required node agent;
• relieve bounded pressure without deleting valuable data;
• cordon and drain according to disruption and storage policy;
• replace the instance through its owning node-pool mechanism.

Treat cordon, drain, node deletion, and reboot as production changes. Confirm the context, workloads, local storage, disruption budgets, distributed-job behavior, and rollback path first.

After readiness returns, verify allocatable resources, device plugins, networking, volumes, replacement Pods, and application outcomes. A green node row is not enough.

Correlate node failure with Polyaxon runs

Polyaxon platform observability adds service and workload context around Kubernetes node signals. Run metadata shows which operations, resources, code, inputs, logs, metrics, and artifacts were affected.

Use termination and retry policy carefully. A retry is safe only if the application checkpoints durable progress and external side effects are idempotent.

Close the incident by documenting the failed boundary, detection gap, workload impact, recovery decision, and prevention control. A reliable runbook turns NotReady from a vague status into a structured investigation.