I am trying to achieve that my kubernetes cluster should have a validity of 5 years, so I have made my ca.crt, apiserver.crt, kubelet-client.crt, front-proxy.crt of 5 years validity and placed those in /etc/kubernetes/pki.
Also, I have enabled my kubelet with client certificate rotation
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki --feature-gates=RotateKubeletClientCertificate=true"So to verify my cluster is working fine I changed the date on my system to 1 day before 1 year expiration and certificate rotation are done properly
Oct 22 06:00:16 ip-10-0-1-170.ec2.internal kubelet[28887]: I1022 06:00:16.806115 28887 reconciler.go:154] Reconciler: start to sync state
Oct 22 06:00:23 ip-10-0-1-170.ec2.internal kubelet[28887]: I1022 06:00:23.546154 28887 transport.go:126] certificate rotation detected, shutting down client connections to start using new credentialsBut once my cluster passes one year it starts showing the error on any kubectl get nodes/pods command: "error: You must be logged in to the server (Unauthorized)"
The possible issue I can think is /etc/kubernetes/admin.conf has only one-year validity certificates. Thanks for your help
Your client-certificate(/etc/kubernetes/admin.conf) is generated for one year. You can generate your client certificate using following command:
kubeadm alpha phase kubeconfig admin --cert-dir /etc/kubernetes/pki --kubeconfig-dir /etc/kubernetes/I have figured out a way to regenerate new admin.conf certificate before expiry of cluster
Generate admin.key and admin.csr using openssl
openssl genrsa -out admin.key 2048
openssl req -new -key admin.key -out admin.csr -subj "/O=system:masters/CN=kubernetes-admin"Now create CSR in kubernetes using above openssl admin.csr
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: admin_csr
spec:
groups:
- system:authenticated
request: $(cat admin.csr | base64 | tr -d '\n')
usages:
- digital signature
- key encipherment
- client auth
EOF
Now approve the CSR generated using kubectl certificate approve admin_csr
Now extract the admin.crt from approved CSR kubectl get csr admin_csr -o jsonpath='{.status.certificate}' | base64 -d > admin.crt
Now change the current user and context to use the new admin key and certificates.
kubectl config set-credentials kubernetes-admin --client-certificate=/home/centos/certs/admin.crt --client-key=/home/centos/certs/admin.key
kubectl config set-context kubernetes-admin@kubernetes --cluster=kubernetes --user=kubernetes-adminAfter this step your kubeconfig which in my case is /root/.kube/config has new client certificate data and key.
Hope this helps.
verified to work with K8s / kubeadm versions
v1.14.x
Make sure to perform these steps on every control plane node:
to manually regenerate certificates, use the following
# all certs
kubeadm alpha certs renew all
# individual cert
# see `kubeadm alpha certs renew --help` for list
kubeadm alpha certs renew apiserver-kubelet-client
here's a helpful script to automate checking cert expiration, and renew if expired: https://gist.github.com/anapsix/974d6c51c7691af45e33302a704ad72b
to regenerate /etc/kubernetes/admin.conf config, one can use the following command, which is evidently a heavily guarded secret, since I was unable to find any documentation mentioning crucial --org system:masters part
kubeadm alpha kubeconfig user \
--org system:masters \
--client-name kubernetes-admin
kubeadm in K8s v1.15.x has new helpful capabilities.
Make sure to perform these steps on every control plane node:
to check expiration
kubeadm alpha certs check-expirationto renew (for example cert in scheduler.conf)
kubeadm alpha certs renew scheduler.conf