I have an old kubernetes cluster on v1.8.6, using kOps I have been trying to upgrade the server to v1.20.8.
I run into a CA cert/key issue where kOps was failing to rolling update the cluster returning an error x509: certificate signed by unknown authority.
kOps had created a new ca secret, so there were now two secrets with type ca. Looking through github issues I found what sounds similar to what I was experiencing and I removed the old ca cert/key with kOps delete secret .... thinking I would move forward with the new one only.
I didn't take a backup of the key, I did get a copy of the crt from S3 but also from my local .kube/config and the master servers ~/.kube/config has the same certificate-authority-data. I can connect to the cluster again with tls via kubectl, but kOps is having issues. There are no ca .crt or .key files in my kOps state store on S3.
Can I restore the key file somehow?
Can I create a kOps secret in this case without the key file, if I plan on upgrading the cluster anyway?
If you have not rotated the control plane/master nodes, you will find the certificates on there. The location for those files have changed over the years, but you should hopefully find them in /srv/kubernetes/.
Or you can restore them if you have enabled S3 versioning.
kOps will reprovision any lost secrets next time you run kops update cluster --yes. You can stick with the new certificates, but then be aware you are effectively doing a key rotation and may want to follow the remaining steps mentioned in this document.