Our Kubernetes 1.6 cluster had certificates generated when the cluster was built on April 13th, 2017.
On December 13th, 2017, our cluster was upgraded to version 1.8, and new certificates were generated [apparently, an incomplete set of certificates].
On April 13th, 2018, we started seeing this message within our Kubernetes dashboard for api-server:
[authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]
Tried pointing client-certificate & client-key within /etc/kubernetes/kubelet.conf at the certificates generated on Dec 13th [apiserver-kubelet-client.crt and apiserver-kubelet-client.crt], but continue to see the above error.
Tried pointing client-certificate & client-key within /etc/kubernetes/kubelet.conf at different certificates generated on Dec 13th [apiserver.crt and apiserver.crt] (I honestly don't understand the difference between these 2 sets of certs/keys), but continue to see the above error.
Tried pointing client-certificate & client-key within /etc/kubernetes/kubelet.conf at non-existent files, and none of the kube* services would start, with /var/log/syslog complaining about this:
Apr 17 17:50:08 kuber01 kubelet[2422]: W0417 17:50:08.181326 2422 server.go:381] invalid kubeconfig: invalid configuration: [unable to read client-cert /tmp/this/cert/does/not/exist.crt for system:node:node01 due to open /tmp/this/cert/does/not/exist.crt: no such file or directory, unable to read client-key /tmp/this/key/does/not/exist.key for system:node:node01 due to open /tmp/this/key/does/not/exist.key: no such file or directory]
Any advice on how to overcome this error, or even troubleshoot it at a more granular level? Was considering regenerating certificates for api-server (kubeadm alpha phase certs apiserver), based on instructions within https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs ... but not sure if I'd be doing more damage.
Relatively new to Kubernetes, and the gentleman who set this up is not available for consult ... any help is appreciated. Thanks.
Each node within the Kubernetes cluster contains a config file for running kubelet ... /etc/kubernetes/kubelet.conf ... and this file is auto-generated by kubeadm. During this auto-generation, kubeadm uses /etc/kubernetes/ca.key to create a node-specific file, /etc/kubernetes/kubelet.conf, within which are two very important pieces ... client-certificate-data and client-key-data. My original thought process led me to believe that I needed to find the corresponding certificate file & key file, renew those files, convert both to base64, and use those values within kubelet.conf files across the cluster ... this thinking was not correct.
Instead, the fix was to use kubeadm to regenerate kubectl.conf on all nodes, as well as admin.conf, controller-manager.conf, and scheduler.conf on the cluster's master node. You'll need /etc/kubernetes/pki/ca.key on each node in order for your config files to include valid data for client-certificate-data and client-key-data.
Pro tip: make use of the --apiserver-advertise-address parameter to ensure your new config files contain the correct IP address of the node hosting the kube-apiserver service.
I think you need re-generate the apiserver certificate /etc/kubernetes/pki/apiserver.crt you can view current expire date like this.
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
Not Before: Dec 20 14:32:00 2017 GMT
Not After : Dec 20 14:32:00 2018 GMTHere is the steps I used to regenerate the certificates on v1.11.5 cluster. compiled steps from here https://github.com/kubernetes/kubeadm/issues/581
to check all certificate expire date:
find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt#x27;|xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'Renew certificate on Master node.
*) Renew certificate
mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old
mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old
mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old
mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old
kubeadm alpha phase certs apiserver --config /root/kubeadm-kubetest.yaml
kubeadm alpha phase certs apiserver-kubelet-client
kubeadm alpha phase certs front-proxy-client
mv /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.crt.old
mv /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.key.old
kubeadm alpha phase certs apiserver-etcd-client
mv /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.crt.old
mv /etc/kubernetes/pki/etcd/server.key /etc/kubernetes/pki/etcd/server.key.old
kubeadm alpha phase certs etcd-server --config /root/kubeadm-kubetest.yaml
mv /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/healthcheck-client.crt.old
mv /etc/kubernetes/pki/etcd/healthcheck-client.key /etc/kubernetes/pki/etcd/healthcheck-client.key.old
kubeadm alpha phase certs etcd-healthcheck-client --config /root/kubeadm-kubetest.yaml
mv /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/peert.crt.old
mv /etc/kubernetes/pki/etcd/peer.key /etc/kubernetes/pki/etcd/peer.key.old
kubeadm alpha phase certs etcd-peer --config /root/kubeadm-kubetest.yaml
*) Backup old configuration files
mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old
kubeadm alpha phase kubeconfig all --config /root/kubeadm-kubetest.yaml
mv $HOME/.kube/config .$HOMEkube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
chmod 777 $HOME/.kube/config
export KUBECONFIG=.kube/config
Reboot the node and check the logs for etcd, kubeapi and kubelet.
Note: Remember to update your CI/CD job kubeconfig file. If you’re using helm command test that also.
On k8s 1.7 I faced a similar problem (x509 expired error included inside /var/log/kube-apiserver.log) and could not find any certificate expired. We decided to restart only the apiserver docker on the master node. It resolved the problem.
$ sudo docker ps -a | grep apiserver
af99f816c7ec gcr.io/google_containers/kube-apiserver@sha256:53b987e5a2932bdaff88497081b488e3b56af5b6a14891895b08703129477d85 "/bin/sh -c '/usr/loc" 15 months ago Up 19 hours k8s_kube-apiserver_kube-apiserver-ip-xxxxxc_0
40f3a18050c3 gcr.io/google_containers/pause-amd64:3.0 "/pause" 15 months ago Up 15 months k8s_POD_kube-apiserver-ip-xxxc_0
$ sudo docker restart af99f816c7ec
af99f816c7ec
$
This topic is also discussed in:
Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":
kubeadm alpha certs check-expirationkubeadm alpha certs renew command.For Kubernetes v1.14 I find this procedure the most helpful:
$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config