Why Rancher container suddenly started to crash?

5/6/2020

I am using Rancher v2.2.2 for managing k8s cluster in Azure AKS. Everything was fine for about one year. All of a sudden my Rancher container started to restart itself every 10 to 15 seconds and Rancher console was unreachable. I found this in container logs:

***Waiting for server to become available: Get https://localhost:6443/version?timeout=30s: x509: certificate has expired or is not yet valid***

Then I checked localhost certificate inside container and appears that it's just expired. Then I shifted time backward on docker host where is my rancher container and problem disappeared and my rancher was again live. But few weeks after I renewed k8s certificates on Azure and now my Rancher with shifted time refuses to connect to k8s cluster API server because it has certificate which is not yet valid.

Please can someone help me? Any ideas how to renew certificate inside Rancher container ?

Thanks

-- asmoljo
containers
docker
kubernetes
rancher
x509certificate

2 Answers

5/21/2020

I had the same problem, to fix it i had to delete the /var/lib/rancher/management-state folder inside the running container. Obviously, the first thing to be done is make a backup of the container data.

$ docker exec -ti sh -c "rm -rf /var/lib/rancher/management-state"

And then restart the container just to be sure about re creating the folder.

$ docker restart

I hope it helps

-- JuanToroMarty
Source: StackOverflow

5/7/2020

I solved the problem by creating a new 'localhost' certificate inside the rancher container. The new certificate is valid for 10 years :)

-- asmoljo
Source: StackOverflow