What happened: The master node does not show ready anymore. Maybe that happend after an failed update (downloaded kubeadm and kubelet in a way too high version)

s-rtk8s01     Ready                         Node     2y120d   v1.14.1
s-rtk8s02     Ready                         Node     2y173d   v1.14.1
s-rtk8s03     Ready                         Node     2y174d   v1.14.1
s-rtk8s04     Ready                         Node     2y174d   v1.14.1
s-rtk8s05     Ready                         Node     2y174d   v1.14.1
s-rtk8sma01   NotReady,SchedulingDisabled   master   2y174d   v1.14.1

Scheduler does not show (after it was deleted forcefully) up in the list of pods but docker ps shows that the static pods are getting started in the background. in the

NAME                                  READY   STATUS    RESTARTS   AGE
coredns-fb8b8dccf-hvh6b               1/1     Running   56         288d
coredns-fb8b8dccf-x5r5h               1/1     Running   58         302d
etcd-s-rtk8sma01                      1/1     Running   45         535d
kube-apiserver-s-rtk8sma01            1/1     Running   13         535d
kube-controller-manager-s-rtk8sma01   1/1     Running   7          485d
kube-flannel-ds-2fmj4                 1/1     Running   6          485d
kube-flannel-ds-5g47f                 1/1     Running   5          485d
kube-flannel-ds-5k27n                 1/1     Running   5          485d
kube-flannel-ds-cj967                 1/1     Running   8          485d
kube-flannel-ds-drjff                 1/1     Running   9          485d
kube-flannel-ds-v4sfg                 1/1     Running   5          485d
kube-proxy-6ngn6                      1/1     Running   11         535d
kube-proxy-85g6c                      1/1     Running   10         535d
kube-proxy-gd5jb                      1/1     Running   13         535d
kube-proxy-grvsk                      1/1     Running   11         535d
kube-proxy-lpht9                      1/1     Running   13         535d
kube-proxy-pmdmj                      0/1     Pending   0          25h

systemd logs for kubelet shows following (I see those errors with the hostname case remarks and an error with a missing mirror pod - maybe the scheduler?)

kubelet_node_status.go:94] Unable to register node "s-rtk8sma01" with API server: nodes "s-rtk8sma01" is forbidden: node "S-RTK8SMA01" is not allowed to modify node "s-rtk8sma01"


setters.go:739] Error getting volume limit for plugin kubernetes.io/azure-disk
setters.go:739] Error getting volume limit for plugin kubernetes.io/cinder
setters.go:739] Error getting volume limit for plugin kubernetes.io/aws-ebs
setters.go:739] Error getting volume limit for plugin kubernetes.io/gce-pd

Generated UID "56ba6ffcb6b23178170f8063052292ee" pod "kube-scheduler" from /etc/kubernetes/manifests/kube-scheduler.yaml
Generated Name "kube-scheduler-s-rtk8sma01" for UID "56ba6ffcb6b23178170f8063052292ee" from URL /etc/kubernetes/manifests/kube-scheduler.yaml
Using namespace "kube-system" for pod "kube-scheduler-s-rtk8sma01" from /etc/kubernetes/manifests/kube-scheduler.yaml
Reading config file "/etc/kubernetes/manifests/kube-scheduler.yaml_bck"
Generated UID "56ba6ffcb6b23178170f8063052292ee" pod "kube-scheduler" from /etc/kubernetes/manifests/kube-scheduler.yaml_bck
Generated Name "kube-scheduler-s-rtk8sma01" for UID "56ba6ffcb6b23178170f8063052292ee" from URL /etc/kubernetes/manifests/kube-scheduler.yaml_bck
Using namespace "kube-system" for pod "kube-scheduler-s-rtk8sma01" from /etc/kubernetes/manifests/kube-scheduler.yaml_bck
Setting pods for source file
anager.go:445] Static pod "56ba6ffcb6b23178170f8063052292ee" (kube-scheduler-s-rtk8sma01/kube-system) does not have a corresponding mirror pod; skipping
anager.go:464] Status Manager: syncPod in syncbatch. pod UID: "24db95fbbd2e618dc6ed589132ed7158"

docker ps shows

aec23e01ee2a        2c4adeb21b4f           "etcd --advertise-cl…"   7 hours ago         Up 7 hours                              k8s_etcd_etcd-s-rtk8sma01_kube-system_24db95fbbd2e618dc6ed589132ed7158_59
97910491f3b2        20a2d7035165           "/usr/local/bin/kube…"   26 hours ago        Up 26 hours                             k8s_kube-proxy_kube-proxy-pmdmj_kube-system_3e807b5e-041d-11eb-a61a-001dd8b72689_0
37d87cdd8886        k8s.gcr.io/pause:3.1   "/pause"                 26 hours ago        Up 26 hours                             k8s_POD_kube-proxy-pmdmj_kube-system_3e807b5e-041d-11eb-a61a-001dd8b72689_0
83a8af0407e5        cfaa4ad74c37           "kube-apiserver --ad…"   39 hours ago        Up 39 hours                             k8s_kube-apiserver_kube-apiserver-s-rtk8sma01_kube-system_57d405cdab537a9a32ce375f1242e4b5_1
85250c421db4        k8s.gcr.io/pause:3.1   "/pause"                 39 hours ago        Up 39 hours                             k8s_POD_kube-apiserver-s-rtk8sma01_kube-system_57d405cdab537a9a32ce375f1242e4b5_1
984a3628068c        3fa2504a839b           "kube-scheduler --bi…"   40 hours ago        Up 40 hours                             k8s_kube-scheduler_kube-scheduler-s-rtk8sma01_kube-system_56ba6ffcb6b23178170f8063052292ee_7
4d5446906cc5        efb3887b411d           "kube-controller-man…"   40 hours ago        Up 40 hours                             k8s_kube-controller-manager_kube-controller-manager-s-rtk8sma01_kube-system_ffbb7c0e6913f72111f95f08ad36e944_3
544423226bed        k8s.gcr.io/pause:3.1   "/pause"                 40 hours ago        Up 40 hours                             k8s_POD_kube-scheduler-s-rtk8sma01_kube-system_56ba6ffcb6b23178170f8063052292ee_4
a75feece56b5        k8s.gcr.io/pause:3.1   "/pause"                 2 days ago          Up 2 days                               k8s_POD_etcd-s-rtk8sma01_kube-system_24db95fbbd2e618dc6ed589132ed7158_20
1b17cb3ef1c1        k8s.gcr.io/pause:3.1   "/pause"                 2 days ago          Up 2 days                               k8s_POD_kube-controller-manager-s-rtk8sma01_kube-system_ffbb7c0e6913f72111f95f08ad36e944_0
c7c7235ed0dc        ff281650a721           "/opt/bin/flanneld -…"   2 months ago        Up 2 months                             k8s_kube-flannel_kube-flannel-ds-v4sfg_kube-system_bc432e78-878f-11e9-9c4b-001dd8b72689_8
d56fe3708565        k8s.gcr.io/pause:3.1   "/pause"                 2 months ago        Up 2 months                             k8s_POD_kube-flannel-ds-v4sfg_kube-system_bc432e78-878f-11e9-9c4b-001dd8b72689_7

What you expected to happen: The master is getting ready again, and the static pods and daemonsets are generated again, so I can start to upgrade the cluster How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: I am really lost at this point and tried many hours to find a solution by myself and hope to get a little bit help from the experts, to understand the problem any maybe get some kind of workaround.

Environment:

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: OnPremise

• OS (e.g: cat /etc/os-release):

  NAME="Ubuntu" VERSION="18.04.2 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.2 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

• Kernel (e.g. uname -a): Linux S-RTK8SMA01 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:
• Network plugin and version (if this is a network-related bug): flannel quay.io/coreos/flannel:v0.11.0-amd64

Does anybody know how to fix those mirror pod problems and knows how I can fix the problem with the node name casing?

What I tried so far was, that I started kubelet with hostname override but this did not have any effect.