Here are my 3 VMs with nodes. I'm not sure when exactly it broke, but I'm assuming right when my VMs shut down and I had to power them back on.
NAME STATUS ROLES AGE VERSION
tjordy-k8-master.myipname NotReady master 99d v1.17.1
tjordy-k8-worker1.myipname NotReady <none> 99d v1.17.1
tjordy-k8-worker2.myipname NotReady <none> 99d v1.17.1
On of the main affects of this is when I try and get logs from a pod or port-forward a pod, I get a connection error.
error: error upgrading connection: error dialing backend: dial tcp 10.18.223.95:10250: connect: no route to host
Here is the describe from my master node:
Name: tjordy-k8-master.myipname
Roles: master
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=tjordy-k8-master.myipname
kubernetes.io/os=linux
node-role.kubernetes.io/master=
Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
node.alpha.kubernetes.io/ttl: 0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Thu, 14 May 2020 02:23:12 -0700
Taints: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/unreachable:NoSchedule
Unschedulable: false
Lease:
HolderIdentity: tjordy-k8-master.myipname
AcquireTime: <unset>
RenewTime: Sat, 15 Aug 2020 11:51:16 -0700
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Thu, 14 May 2020 14:13:24 -0700 Thu, 14 May 2020 14:13:24 -0700 WeaveIsUp Weave pod has set this
MemoryPressure Unknown Sat, 15 Aug 2020 11:50:47 -0700 Fri, 21 Aug 2020 16:03:03 -0700 NodeStatusUnknown Kubelet stopped posting node status.
DiskPressure Unknown Sat, 15 Aug 2020 11:50:47 -0700 Fri, 21 Aug 2020 16:03:03 -0700 NodeStatusUnknown Kubelet stopped posting node status.
PIDPressure Unknown Sat, 15 Aug 2020 11:50:47 -0700 Fri, 21 Aug 2020 16:03:03 -0700 NodeStatusUnknown Kubelet stopped posting node status.
Ready Unknown Sat, 15 Aug 2020 11:50:47 -0700 Fri, 21 Aug 2020 16:03:03 -0700 NodeStatusUnknown Kubelet stopped posting node status.
Addresses:
InternalIP: 10.18.223.22
Hostname: tjordy-k8-master.myipname
Capacity:
cpu: 2
ephemeral-storage: 100112644Ki
hugepages-2Mi: 0
memory: 3880788Ki
pods: 110
Allocatable:
cpu: 2
ephemeral-storage: 92263812558
hugepages-2Mi: 0
memory: 3778388Ki
pods: 110
System Info:
Machine ID: b116c790be914ec08657e4cc260f0164
System UUID: 4216A453-81C5-3477-2710-CF356A1B0BFE
Boot ID: c73333b0-cd1c-40f2-8877-28a8a4b4bd05
Kernel Version: 3.10.0-957.10.1.el7.x86_64
OS Image: CentOS Linux 7 (Core)
Operating System: linux
Architecture: amd64
Container Runtime Version: docker://19.3.8
Kubelet Version: v1.17.1
Kube-Proxy Version: v1.17.1
Non-terminated Pods: (9 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE
--------- ---- ------------ ---------- --------------- ------------- ---
default prometheus-operator-1589787700-prometheus-node-exporter-nb4fv 0 (0%) 0 (0%) 0 (0%) 0 (0%) 95d
kube-system coredns-6955765f44-pk8jm 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 99d
kube-system coredns-6955765f44-xkfk6 100m (5%) 0 (0%) 70Mi (1%) 170Mi (4%) 99d
kube-system etcd-tjordy-k8-master.myipname 0 (0%) 0 (0%) 0 (0%) 0 (0%) 99d
kube-system kube-apiserver-tjordy-k8-master.myipname 250m (12%) 0 (0%) 0 (0%) 0 (0%) 99d
kube-system kube-controller-manager-tjordy-k8-master.myipname 200m (10%) 0 (0%) 0 (0%) 0 (0%) 99d
kube-system kube-proxy-xcg6h 0 (0%) 0 (0%) 0 (0%) 0 (0%) 99d
kube-system kube-scheduler-tjordy-k8-master.myipname 100m (5%) 0 (0%) 0 (0%) 0 (0%) 99d
kube-system weave-net-6fmv6 20m (1%) 0 (0%) 0 (0%) 0 (0%) 99d
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 770m (38%) 0 (0%)
memory 140Mi (3%) 340Mi (9%)
ephemeral-storage 0 (0%) 0 (0%)
Events: <none>
I've tried restarting them but nothing changed. Not really sure what to do.
Edit: Ran journalctl -u kubelet. Theres thousands of lines but are mostly characterized by
Aug 21 19:44:42 3nxdomain kubelet[8060]: I0821 19:44:42.158482 8060 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.189131 8060 kubelet.go:2263] node "3nxdomain" not found
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.289338 8060 kubelet.go:2263] node "3nxdomain" not found
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.390437 8060 kubelet.go:2263] node "3nxdomain" not found
Aug 21 19:44:42 3nxdomain kubelet[8060]: I0821 19:44:42.411680 8060 kubelet_node_status.go:70] Attempting to register node 3nxdomain
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.413954 8060 kubelet_node_status.go:92] Unable to register node "3nxdomain" with API server: nodes "3nxdomain" is forbidden: node "tjordy-k8-master.myip" is not allowed to modify node "3nxdomain"
Aug 21 19:44:42 3nxdomain kubelet[8060]: E0821 19:44:42.490625 8060 kubelet.go:2263] node "3nxdomain" not found