K
Q

Renew kubernetes pki after expired

5/27/2019

My kubernetes PKI expired (API server to be exact) and I can't find a way to renew it. The error I get is

May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922595    8751 server.go:417] Version: v1.14.2
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922784    8751 plugins.go:103] No cloud provider specified.
May 27 08:43:51 node1 kubelet[8751]: I0527 08:43:51.922800    8751 server.go:754] Client rotation is on, will bootstrap in background
May 27 08:43:51 node1 kubelet[8751]: E0527 08:43:51.925859    8751 bootstrap.go:264] Part of the existing bootstrap client certificate is expired: 2019-05-24 13:24:42 +0000 UTC
May 27 08:43:51 node1 kubelet[8751]: F0527 08:43:51.925894    8751 server.go:265] failed to run Kubelet: unable to load bootstrap
kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory

The documentation on https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/ describes how to renew but it only works if the API server is not expired. I have tried to do a

kubeadm alpha cert renew all

and do a reboot but that just made the entire cluster fail so I did a rollback to a snapshot (my cluster is running on VMware).

The cluster is running and all containers seem to work but I can't access it via kubectl so I can't really deploy or query.

My kubernetes version is 1.14.2.

-- Kim Nielsen
kubernetes
ssl-certificate

Similar Questions

How Alb Ingress Controller target-type:instance forward requests
How to run groovy scripts on slave node by passing the groovy script file path in jenkins
Moving Statefulsets with PV between K8s clusters
What URLs does the GKE extensible service proxy need to access
How to check readiness without creating a deadlock in a cluster configuration in Kubernetes?
Exception occurred while authenticating Firebase
Image pull over multiple K8s nodes

5 Answers

2/25/2020

I use a config.yaml to configure the Masters so for me, the answer was:

sudo -i
mkdir -p ~/k8s_backup/etcd
cd /etc/kubernetes/pki/
mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/k8s_backup
cd /etc/kubernetes/pki/etcd
mv {healthcheck-client.crt,healthcheck-client.key,peer.crt,peer.key,server.crt,server.key} ~/k8s_backup/etcd/
kubeadm init phase certs all --ignore-preflight-errors=all --config /etc/kubernetes/config.yaml

cd /etc/kubernetes
mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/k8s_backup
kubeadm init phase kubeconfig all --config /etc/kubernetes/config.yaml --ignore-preflight-errors=all

For good measure I reboot

shutdown now -r
-- Max
Source: StackOverflow
5/28/2019

So the solution was to (first a backup)

$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot

then

$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

that did the job for me and thanks for your hints :)

-- Kim Nielsen
Source: StackOverflow
8/1/2019

This topic is also discussed in:

Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":

kubeadm alpha certs check-expiration
  • Automatic certificate renewal:
    • kubeadm renews all the certificates during control plane upgrade.
  • Manual certificate renewal:
    • You can renew your certificates manually at any time with the kubeadm alpha certs renew command.
    • This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.

Overall for Kubernetes v1.14 I find this procedure the most helpful:

-- Tomasz Tarczynski
Source: StackOverflow
5/27/2019

Try to do cert renewal via kubeadm init phase certs command.

You can check certs expiration via the following command:

openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text

openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -text

First, ensure that you have most recent backup of k8s certificates inventory /etc/kubernetes/pki/*.

Delete apiserver.* and apiserver-kubelet-client.* cert files in /etc/kubernetes/pki/ directory.

Spawn a new certificates via kubeadm init phase certs command:

sudo kubeadm init phase certs apiserver

sudo kubeadm init phase certs apiserver-kubelet-client

Restart kubelet and docker daemons:

sudo systemctl restart docker; sudo systemctl restart kubelet

You can find more related information in the official K8s documentation.

-- mk_sta
Source: StackOverflow
3/16/2020
[root@nrchbs-slp4115 ~]# kubectl get apiservices |egrep metrics
v1beta1.metrics.k8s.io                 kube-system/metrics-server   True        125m


[root@nrchbs-slp4115 ~]# kubectl get svc -n kube-system
NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns         ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   20d
metrics-server   ClusterIP   10.99.2.11   <none>        443/TCP                  125m


[root@nrchbs-slp4115 ~]# kubectl get ep -n kube-system
NAME                      ENDPOINTS                                               AGE
kube-controller-manager   <none>                                                  20d
kube-dns                  10.244.0.5:53,10.244.0.6:53,10.244.0.5:53 + 3 more...   20d
kube-scheduler            <none>                                                  20d
metrics-server            10.244.2.97:443                                         125m
[root@nrchbs-slp4115 ~]#
-- sanjay singh
Source: StackOverflow