During Kubernetes upgrade from 1.14 to 1.15 with kubespray, my team had a blocking issue with the following message during the "Upgrade first master" step :
[upgrade/apply] FATAL: couldn''t upgrade control plane.
kubeadm has tried to recover everything into the earlier state.
Errors faced: [failed to renew certificates for component "kube-apiserver":
failed to renew certificate apiserver-kubelet-client:
unable to sign certificate:
must specify at least one ExtKeyUsage,
rename /etc/kubernetes/tmp/kubeadm
-backup-manifests-2019-09-19-09-06-27/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml: no such file or directory]'Trying to isolate the task and manually running the kubeadm command line leads to the same error message :
#/usr/local/bin/kubeadm upgrade apply -y v1.15.3 --config=/etc/kubernetes/kubeadm-config.yaml --ignore-preflight-errors=all --allow-experimental-upgrades --allow-release-candidate-upgrades --etcd-upgrade=false -v 6 or even trying to just manually renew the certificate :
/etc/kubernetes/pki# kubeadm alpha certs renew apiserver-kubelet-client -v 9
I0919 14:42:11.515503 18597 initconfiguration.go:105] detected and using CRI socket: /var/run/dockershim.sock
I0919 14:42:11.515897 18597 interface.go:384] Looking for default routes with IPv4 addresses
I0919 14:42:11.515916 18597 interface.go:389] Default route transits interface “eth0”
I0919 14:42:11.516284 18597 interface.go:196] Interface eth0 is up
(...)
I0919 14:42:11.516835 18597 feature_gate.go:216] feature gates: &{map[]}
failed to renew certificate apiserver-kubelet-client: unable to sign certificate: must specify at least one ExtKeyUsage
The solution was found eventually and posted below.
The issue comes from kubeadm which uses the old certificates when it has to renew them. But when these initial certificates are too old or were manually generated, they may not include some mandatory fields that needs to be there.
In the error message, ExtKeyUsage refers to the X509v3 Extended Key Usage field.
You can check that by looking into your certificates : 2 certificates are involved : apiserver-kubelet-client.crt and front-proxy-client.crt
They are located on the master hosts at /etc/kubernetes/pki.
You can check them with
# openssl x509 -in apiserver-kubelet-client.crt -text -noout
If they do not contain the following (near the end), then kubeadm will be totally unable to renew the certificate
(...)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client AuthenticationTL;DR;
The solution is simply to create brand new certificates with the following procedure
######### Backup your certificates (just in case)
master01:/etc/kubernetes/pki# cp -a /etc/kubernetes/pki /root/backup_cert/
######### Delete incomplete certificates
master01:/etc/kubernetes/pki# rm apiserver-kubelet-client.*
master01:/etc/kubernetes/pki# rm front-proxy-client.*
######### Then recreate them
master01:/etc/kubernetes/pki# kubeadm init phase certs apiserver-kubelet-client
master01:/etc/kubernetes/pki# kubeadm init phase certs front-proxy-client
You can now restart your upgrade procedure which should be ok. (Note : if your cluster was left in a state where your first master has a SchedulingDisabled state, then do not forget to uncordon the host, because kubespray playbook won't fix that)