kubernetes worker node in "NotReady" status

1/7/2019

I am trying to setup my first cluster using Kubernetes 1.13.1. The master got initialized okay, but both of my worker nodes are NotReady. kubectl describe node shows that Kubelet stopped posting node status on both worker nodes. On one of the worker nodes I get log output like

> kubelet[3680]: E0107 20:37:21.196128    3680 kubelet.go:2266] node
> "xyz" not found.

Here is the full details:

I am using Centos 7 & Kubernetes 1.13.1.

Initializing was done as follows:

[root@master ~]# kubeadm init --apiserver-advertise-address=10.142.0.4 --pod-network-cidr=10.142.0.0/24

Successfully initialized the cluster:

You can now join any number of machines by running the following on each node
as root:
`kubeadm join 10.142.0.4:6443 --token y0epoc.zan7yp35sow5rorw --discovery-token-ca-cert-hash sha256:f02d43311c2696e1a73e157bda583247b9faac4ffb368f737ee9345412c9dea4`

deployed the flannel CNI:

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

The join command worked fine.

[kubelet-start] Activating the kubelet service [tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap... [patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node01" as an annotation

This node has joined the cluster:

* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the master to see this node join the cluster.

Result of kubectl get nodes:

[root@master ~]# kubectl get nodes

NAME     STATUS     ROLES    AGE   VERSION

master   Ready      master   9h    v1.13.1

node01   NotReady   <none>   9h    v1.13.1

node02   NotReady   <none>   9h    v1.13.1

on both nodes:

[root@node01 ~]# service kubelet status Redirecting to /bin/systemctl status kubelet.service ● kubelet.service - kubelet: The Kubernetes Node Agent

 Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)

  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf

   Active: active (running) since Tue 2019-01-08 04:49:20 UTC; 32s ago

     Docs: https://kubernetes.io/docs/

 Main PID: 4224 (kubelet)

   Memory: 31.3M

   CGroup: /system.slice/kubelet.service
           └─4224 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfi

`Jan 08 04:54:10 node01 kubelet[4224]: E0108 04:54:10.957115    4224 kubelet.go:2266] node "node01" not found`

I appreciate your advise on how to troubleshoot this.

-- Falcon
flannel
kubeadm
kubernetes

2 Answers

7/18/2019

The previous answer sounds correct. You can verify that by running kubectl describe node node01 on the master, or wherever kubectl is correctly configured.

-- Erik Martensen
Source: StackOverflow

1/8/2019

It seems like the reason of this error is due to incorrect subnet. In Flannel documentation it is written that you should use /16 not /24 for pod network.

NOTE: If kubeadm is used, then pass --pod-network-cidr=10.244.0.0/16 to kubeadm init to ensure that the podCIDR is set.

I tried to run kubeadm with /24 and although I had nodes in Ready state the flannel pods did not run properly which resulted in some issues.

You can check if your flannel pods are running properly by: kubectl get pods -n kube-system if the status is other than running then it is incorrect behavior. In this case you can check details by running kubectl describe pod PODNAME -n kube-system. Try changing the subnet and update us if that fixed the problem.

-- aurelius
Source: StackOverflow