K
Q

How to debug when Kubernetes nodes are in 'Not Ready' state

11/4/2017

I initialized the master node and add 2 worker nodes, but only master and one of the worker node show up when I run the following command:

kubectl get nodes

also, both these nodes are in 'Not Ready' state. What are the steps should I take to understand what the problem could be?

  • I can ping all the nodes from each of the other nodes.
  • The version of Kubernetes is 1.8.
  • OS is Cent OS 7

  • I used the following repo to install Kubernetes:

    cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes] name=Kubernetes
baseurl=http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
EOF

yum install kubelet kubeadm kubectl kubernetes-cni
-- lex
kubernetes

Similar Questions

cassandra stateful set in kubernetes
Kubernetes executor gitlab
use a customer-encrypted disk in a kubernetes pod
Azure Log Analytics/OMS - reconstruct container's log as raw text

3 Answers

4/18/2019

I was having similar issue because of a different reason:

Error:

cord@node1:~$ kubectl get nodes
NAME    STATUS     ROLES    AGE     VERSION
node1   Ready      master   17h     v1.13.5
node2   Ready      <none>   17h     v1.13.5
node3   NotReady   <none>   9m48s   v1.13.5

cord@node1:~$ kubectl describe node node3
Name:               node3
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  Ready            False   Thu, 18 Apr 2019 01:15:46 -0400   Thu, 18 Apr 2019 01:03:48 -0400   KubeletNotReady              runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Addresses:
  InternalIP:  192.168.2.6
  Hostname:    node3

cord@node3:~$ journalctl -u kubelet

Apr 18 01:24:50 node3 kubelet[54132]: W0418 01:24:50.649047   54132 cni.go:149] Error loading CNI config list file /etc/cni/net.d/10-calico.conflist: error parsing configuration list: no 'plugins' key
Apr 18 01:24:50 node3 kubelet[54132]: W0418 01:24:50.649086   54132 cni.go:203] Unable to update cni config: No valid networks found in /etc/cni/net.d
Apr 18 01:24:50 node3 kubelet[54132]: E0418 01:24:50.649402   54132 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 18 01:24:55 node3 kubelet[54132]: W0418 01:24:55.650816   54132 cni.go:149] Error loading CNI config list file /etc/cni/net.d/10-calico.conflist: error parsing configuration list: no 'plugins' key
Apr 18 01:24:55 node3 kubelet[54132]: W0418 01:24:55.650845   54132 cni.go:203] Unable to update cni config: No valid networks found in /etc/cni/net.d
Apr 18 01:24:55 node3 kubelet[54132]: E0418 01:24:55.651056   54132 kubelet.go:2192] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Apr 18 01:24:57 node3 kubelet[54132]: I0418 01:24:57.248519   54132 setters.go:72] Using node IP: "192.168.2.6"

Issue:

My file: 10-calico.conflist was incorrect. Verified it from a different node and from sample file in the same directory "calico.conflist.template".

Resolution:

Changing the file, "10-calico.conflist" and restarting the service using "systemctl restart kubelet", resolved my issue:

NAME    STATUS   ROLES    AGE   VERSION
node1   Ready    master   18h   v1.13.5
node2   Ready    <none>   18h   v1.13.5
node3   Ready    <none>   48m   v1.13.5
-- Syed Faraz Umar
Source: StackOverflow
7/19/2019

I found applying the network and rebooting both the nodes did the trick for me.

kubectl apply -f [podnetwork].yaml

-- ReadyPlayer1
Source: StackOverflow
11/4/2017

First, describe nodes and see if it reports anything:

$ kubectl describe nodes

Look for conditions, capacity and allocatable:

Conditions:
  Type              Status
  ----              ------
  OutOfDisk         False
  MemoryPressure    False
  DiskPressure      False
  Ready             True
Capacity:
 cpu:       2
 memory:    2052588Ki
 pods:      110
Allocatable:
 cpu:       2
 memory:    1950188Ki
 pods:      110

If everything is alright here, SSH into the node and observe kubelet logs to see if it reports anything. Like certificate erros, authentication errors etc.

If kubelet is running as a systemd service, you can use

$ journalctl -u kubelet

-- Shahidh
Source: StackOverflow