Kubelet nodes set to NotReady after my VMs restarted

8/22/2020

Here are my 3 VMs with nodes. I'm not sure when exactly it broke, but I'm assuming right when my VMs shut down and I had to power them back on.

NAME                            STATUS     ROLES    AGE   VERSION
tjordy-k8-master.myipname       NotReady   master   99d   v1.17.1
tjordy-k8-worker1.myipname      NotReady   <none>   99d   v1.17.1
tjordy-k8-worker2.myipname      NotReady   <none>   99d   v1.17.1

On of the main affects of this is when I try and get logs from a pod or port-forward a pod, I get a connection error.

error: error upgrading connection: error dialing backend: dial tcp 10.18.223.95:10250: connect: no route to host

Here is the describe from my master node:

Name:               tjordy-k8-master.myipname
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=tjordy-k8-master.myipname
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 14 May 2020 02:23:12 -0700
Taints:             node-role.kubernetes.io/master:NoSchedule
                    node.kubernetes.io/unreachable:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  tjordy-k8-master.myipname
  AcquireTime:     <unset>
  RenewTime:       Sat, 15 Aug 2020 11:51:16 -0700
Conditions:
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----                 ------    -----------------                 ------------------                ------              -------
  NetworkUnavailable   False     Thu, 14 May 2020 14:13:24 -0700   Thu, 14 May 2020 14:13:24 -0700   WeaveIsUp           Weave pod has set this
  MemoryPressure       Unknown   Sat, 15 Aug 2020 11:50:47 -0700   Fri, 21 Aug 2020 16:03:03 -0700   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure         Unknown   Sat, 15 Aug 2020 11:50:47 -0700   Fri, 21 Aug 2020 16:03:03 -0700   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure          Unknown   Sat, 15 Aug 2020 11:50:47 -0700   Fri, 21 Aug 2020 16:03:03 -0700   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready                Unknown   Sat, 15 Aug 2020 11:50:47 -0700   Fri, 21 Aug 2020 16:03:03 -0700   NodeStatusUnknown   Kubelet stopped posting node status.
Addresses:
  InternalIP:  10.18.223.22
  Hostname:    tjordy-k8-master.myipname
Capacity:
  cpu:                2
  ephemeral-storage:  100112644Ki
  hugepages-2Mi:      0
  memory:             3880788Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  92263812558
  hugepages-2Mi:      0
  memory:             3778388Ki
  pods:               110
System Info:
  Machine ID:                 b116c790be914ec08657e4cc260f0164
  System UUID:                4216A453-81C5-3477-2710-CF356A1B0BFE
  Boot ID:                    c73333b0-cd1c-40f2-8877-28a8a4b4bd05
  Kernel Version:             3.10.0-957.10.1.el7.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.8
  Kubelet Version:            v1.17.1
  Kube-Proxy Version:         v1.17.1
Non-terminated Pods:          (9 in total)
  Namespace                   Name                                                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                               ------------  ----------  ---------------  -------------  ---
  default                     prometheus-operator-1589787700-prometheus-node-exporter-nb4fv      0 (0%)        0 (0%)      0 (0%)           0 (0%)         95d
  kube-system                 coredns-6955765f44-pk8jm                                           100m (5%)     0 (0%)      70Mi (1%)        170Mi (4%)     99d
  kube-system                 coredns-6955765f44-xkfk6                                           100m (5%)     0 (0%)      70Mi (1%)        170Mi (4%)     99d
  kube-system                 etcd-tjordy-k8-master.myipname                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         99d
  kube-system                 kube-apiserver-tjordy-k8-master.myipname             250m (12%)    0 (0%)      0 (0%)           0 (0%)         99d
  kube-system                 kube-controller-manager-tjordy-k8-master.myipname    200m (10%)    0 (0%)      0 (0%)           0 (0%)         99d
  kube-system                 kube-proxy-xcg6h                                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         99d
  kube-system                 kube-scheduler-tjordy-k8-master.myipname             100m (5%)     0 (0%)      0 (0%)           0 (0%)         99d
  kube-system                 weave-net-6fmv6                                                    20m (1%)      0 (0%)      0 (0%)           0 (0%)         99d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                770m (38%)  0 (0%)
  memory             140Mi (3%)  340Mi (9%)
  ephemeral-storage  0 (0%)      0 (0%)
Events:              <none>

I've tried restarting them but nothing changed. Not really sure what to do.

Edit: Ran journalctl -u kubelet. Theres thousands of lines but are mostly characterized by

Aug 21 19:44:42 3nxdomain kubelet[8060]: I0821 19:44:42.158482    8060 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.189131    8060 kubelet.go:2263] node "3nxdomain" not found
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.289338    8060 kubelet.go:2263] node "3nxdomain" not found
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.390437    8060 kubelet.go:2263] node "3nxdomain" not found
Aug 21 19:44:42 3nxdomain kubelet[8060]: I0821 19:44:42.411680    8060 kubelet_node_status.go:70] Attempting to register node 3nxdomain
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.413954    8060 kubelet_node_status.go:92] Unable to register node "3nxdomain" with API server: nodes "3nxdomain" is forbidden: node "tjordy-k8-master.myip" is not allowed to modify node "3nxdomain"
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.490625    8060 kubelet.go:2263] node "3nxdomain" not found
-- Trevor Jordy
kubelet
kubernetes

0 Answers