Suddenly my kubernetes cluster shows this error when I deployment a new service into this cluster:
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "02ceaa702704151a69b1f9f5210d4a68956c2b41050b711319f07d38f57364af": error getting ClusterInformation: Get "https://10.96.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default": x509: certificate signed by unknown authority
This cluster running fine for monthes, I have using this command to check my kubernetes cluster certificate like this:
[root@k8smasterone ~]# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
Not Before: Aug 3 13:31:21 2021 GMT
Not After : Aug 4 04:40:53 2022 GMT
the certificate did not expire util 2022.08, why shows this error? what should I do to fix this problem? This is all the certificate expire status:
[root@k8smasterone ~]# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Aug 03, 2022 13:31 UTC 234d no
apiserver Aug 04, 2022 04:40 UTC 234d ca no
apiserver-etcd-client Aug 03, 2022 13:31 UTC 234d etcd-ca no
apiserver-kubelet-client Aug 03, 2022 13:31 UTC 234d ca no
controller-manager.conf Aug 03, 2022 13:31 UTC 234d no
etcd-healthcheck-client Aug 03, 2022 13:31 UTC 234d etcd-ca no
etcd-peer Aug 03, 2022 13:31 UTC 234d etcd-ca no
etcd-server Aug 03, 2022 13:31 UTC 234d etcd-ca no
front-proxy-client Aug 03, 2022 13:31 UTC 234d front-proxy-ca no
scheduler.conf Aug 03, 2022 13:31 UTC 234d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Aug 01, 2031 13:31 UTC 9y no
etcd-ca Aug 01, 2031 13:31 UTC 9y no
front-proxy-ca Aug 01, 2031 13:31 UTC 9y no
this is my kubernetes version information:
[root@k8smasterone ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.3", GitCommit:"ca643a4d1f7bfe34773c74f79527be4afd95bf39", GitTreeState:"clean", BuildDate:"2021-07-15T21:04:39Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2", GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean", BuildDate:"2021-06-16T12:53:14Z", GoVersion:"go1.16.5", Compiler:"gc", Platform:"linux/amd64"}
I am sure the kube proxy running in the master node.
Its not the issue with certificate expiry, it is the issue with the trust between the cluster components, seems like that trust is broken. This might be related. Please check the following, if that helps: https://discuss.kubernetes.io/t/solved-x509-certificate-error/14151 https://programmerah.com/failed-to-create-pod-sandbox-rpc-error-code-unknown-desc-failed-to-set-up-sandbox-container-28097/