In some cases after a node reboot the Kubernetes cluster managed by MicroK8S cannot start pods.
With a describe of the pod failing to be ready I could see that the pod was stuck in a "Pulling image" state during several minutes without any other event, as following:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 50s default-scheduler Successfully assigned default/my-service-abc to my-desktop
Normal Pulling 8m kubelet Pulling image "127.0.0.1:32000/some-service"Pulling from the node with docker pull 127.0.0.1:32000/some-service works perfectly, so it doesn't seem a problem with Docker.
I have upgraded Docker just in case.
I seem to run the latest version of microk8s.
Running a sudo microk8s inspect gives no error/warning, everything seemed to work perfectly.
As a Docker pull did work locally, it's really the Kubelet app which is communicating with Docker that seemed stuck.
Even with a sudo service docker stop && sudo service docker start it did not work.
Even a rollout did not suffice to get out of the Pulling state after the reboot of docker.
Worst of all, a reboot of the server did not change anything (the pod that were up were still working, but all the other pods (70%) were down and in ContainerCreating state.
Checking the status systemctl status snap.microk8s.daemon-kubelet did not report any error.
The only thing that worked seemed to do this:
sudo systemctl reboot snap.microk8s.daemon-kubelet
However, it also rebooted the whole node, so that's something to do in a last case scenario (same as the node reboot).