In my v1.23.1
test cluster I see worker node certificate expired some time ago. but worker node still taking the workload and in Ready
status.
How this certificate is getting used, when we will see the issue with expired certificate?
# curl -v https://localhost:10250 -k 2>&1 |grep 'expire date'
* expire date: Oct 4 18:02:14 2021 GMT
# openssl x509 -text -noout -in /var/lib/kubelet/pki/kubelet.crt |grep -A2 'Validity'
Validity
Not Before: Oct 4 18:02:14 2020 GMT
Not After : Oct 4 18:02:14 2021 GMT
Update 1:
Cluster is running on-perm with CentOS Stream 8
OS , build with kubeadm
tool. I was able to schedule the workload on all the worker nodes. created nginx deployment and scaled it 50 pods, I can see nginx PODs on all the worker nodes.
Also I can reboot the work nodes with-out any issue.
Update 2:
kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
W0303 11:17:18.261639 698383 utils.go:69] The recommended value for "resolvConf" in "KubeletConfiguration" is: /run/systemd/resolve/resolv.conf; the provided value is: /run/systemd/resolve/resolv.conf
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jan 16, 2023 16:15 UTC 318d ca no
apiserver Jan 16, 2023 16:15 UTC 318d ca no
apiserver-kubelet-client Jan 16, 2023 16:15 UTC 318d ca no
controller-manager.conf Jan 16, 2023 16:15 UTC 318d ca no
front-proxy-client Jan 16, 2023 16:15 UTC 318d front-proxy-ca no
scheduler.conf Jan 16, 2023 16:15 UTC 318d ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Oct 02, 2030 18:44 UTC 8y no
front-proxy-ca Oct 02, 2030 18:44 UTC 8y no
Thanks
Update 3
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
server10 Ready control-plane,master 519d v1.23.1
server11 Ready control-plane,master 519d v1.23.1
server12 Ready control-plane,master 519d v1.23.1
server13 Ready <none> 519d v1.23.1
server14 Ready <none> 519d v1.23.1
server15 Ready <none> 516d v1.23.1
server16 Ready <none> 516d v1.23.1
server17 Ready <none> 516d v1.23.1
server18 Ready <none> 516d v1.23.1
# kubectl get pods -o wide
nginx-dev-8677c757d4-4k9xp 1/1 Running 0 4d12h 10.203.53.19 server17 <none> <none>
nginx-dev-8677c757d4-6lbc6 1/1 Running 0 4d12h 10.203.89.120 server14 <none> <none>
nginx-dev-8677c757d4-ksckf 1/1 Running 0 4d12h 10.203.124.4 server16 <none> <none>
nginx-dev-8677c757d4-lrz9h 1/1 Running 0 4d12h 10.203.124.41 server16 <none> <none>
nginx-dev-8677c757d4-tllx9 1/1 Running 0 4d12h 10.203.151.70 server11 <none> <none>
# grep client /etc/kubernetes/kubelet.conf
client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem
# ls -ltr /var/lib/kubelet/pki
total 16
-rw------- 1 root root 1679 Oct 4 2020 kubelet.key
-rw-r--r-- 1 root root 2258 Oct 4 2020 kubelet.crt
-rw------- 1 root root 1114 Oct 4 2020 kubelet-client-2020-10-04-14-50-21.pem
lrwxrwxrwx 1 root root 59 Jul 6 2021 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-07-06-01-44-10.pem
-rw------- 1 root root 1114 Jul 6 2021 kubelet-client-2021-07-06-01-44-10.pem