Currently I am using a script to renew Kubernetes certificates before they expire. But this is a manual process. I have to monitor expiration dates carefully and run this script beforehand. What's the recommended way to update all control plane certificates automatically without updating control plane? Do kubelet's --rotate* flags rotate all components (e.g. controller) or it is just for kubelet? PS: Kubernetes cluster was created with kubeadm.
As of kubernetes 1.8 certificate rotation is added. You can read about it here, https://kubernetes.io/docs/tasks/tls/certificate-rotation/
Answering following question:
What's the recommended way to update all control plane certificates automatically without updating control plane
According to the k8s docs and best practices the best practice is to use "Automatic certificate renewal" with control plane upgrade:
Automatic certificate renewal
This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.
Note: It is a best practice to upgrade your cluster frequently in order to stay secure.
-- Kubernetes.io: Administer cluster: Kubeadm certs: Automatic certificate renewal
Why this is the recommended way:
From the best practices standpoint you should be upgrading your control-plane to patch vulnerabilities, add features and use the version that is currently supported.
Each control-plane upgrade will renew the certificates as described (defaults to true):
$ kubeadm upgrade apply --help
--certificate-renewal Perform the renewal of certificates used by component changed during upgrades. (default true)You can also check the expiration of the control-plane certificates by running:
$ kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf May 30, 2022 13:36 UTC 364d no
apiserver May 30, 2022 13:36 UTC 364d ca no
apiserver-etcd-client May 30, 2022 13:36 UTC 364d etcd-ca no
apiserver-kubelet-client May 30, 2022 13:36 UTC 364d ca no
controller-manager.conf May 30, 2022 13:36 UTC 364d no
etcd-healthcheck-client May 30, 2022 13:36 UTC 364d etcd-ca no
etcd-peer May 30, 2022 13:36 UTC 364d etcd-ca no
etcd-server May 30, 2022 13:36 UTC 364d etcd-ca no
front-proxy-client May 30, 2022 13:36 UTC 364d front-proxy-ca no
scheduler.conf May 30, 2022 13:36 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca May 28, 2031 13:36 UTC 9y no
etcd-ca May 28, 2031 13:36 UTC 9y no
front-proxy-ca May 28, 2031 13:36 UTC 9y no
A side note!
kubelet.confis not included in the list above becausekubeadmconfigureskubeletfor automatic certificate renewal.
From what it can be seen by default:
There are other features that allows you to rotate the certificates in a "semi automatic" way.
You can opt for a manual certificate renewal with the:
$ kubeadm certs renew
where you can automatically (with the command) renew the specified (or all) certificates:
$ kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.Please take a specific look on the output:
You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.As pointed, you will need to restart the components of your control-plane to use new certificate but remember:
$ kubectl delete pod -n kube-system kube-scheduler-ubuntu will not work.
You will need to restart the docker container responsible for the component:
$ docker ps | grep -i "scheduler"$ docker restart 8c361562701b (example)
8c361562701b 38f903b54010 "kube-scheduler --au…" 11 minutes ago Up 11 minutes k8s_kube-scheduler_kube-scheduler-ubuntu_kube-system_dbb97c1c9c802fa7cf2ad7d07938bae9_5
b709e8fb5e6c k8s.gcr.io/pause:3.4.1 "/pause" About an hour ago Up About an hour k8s_POD_kube-scheduler-ubuntu_kube-system_dbb97c1c9c802fa7cf2ad7d07938bae9_0As pointed in below link, kubelet can automatically renew it's certificate (kubeadm configures the cluster in a way that this option is enabled):
Depending on the version used in your environment, this can be disabled. Currently in the newest version of k8s managed by kubeadm this option is enabled by default according to my knowledge.
Please keep in mind that before you start with any kubernetes node/control plane/update/upgrade to read "Urgent Upgrade Notes" specific to your k8s version (example):
Defining the automatic way of certificate rotation could go in either way but you can use already mentioned commands to automate this process. You would need to create a script (which you already have) that would be put in cron that would fire after some time and renew them.