K
Q

kubeadm init shows kubelet isn't running or healthy

8/31/2018

I am trying to run Kubernetes and trying to use sudo kubeadm init. Swap is off as recommended by official doc.

The issue is it displays the warning:

[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.


Unfortunately, an error has occurred:
            timed out waiting for the condition

This error is likely caused by:
            - The kubelet is not running
            - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)
            - No internet connection is available so the kubelet cannot pull or find the following control plane images:
                - k8s.gcr.io/kube-apiserver-amd64:v1.11.2
                - k8s.gcr.io/kube-controller-manager-amd64:v1.11.2
                - k8s.gcr.io/kube-scheduler-amd64:v1.11.2
                - k8s.gcr.io/etcd-amd64:3.2.18
                - You can check or miligate this in beforehand with "kubeadm config images pull" to make sure the images
                  are downloaded locally and cached.

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
            - 'systemctl status kubelet'
            - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI, e.g. docker.
        Here is one example how you may list all Kubernetes containers running in docker:
            - 'docker ps -a | grep kube | grep -v pause'
            Once you have found the failing container, you can inspect its logs with:
            - 'docker logs CONTAINERID'
couldn't initialize a Kubernetes cluster

The docker version I am using is Docker version 17.03.2-ce, build f5ec1e2 I m using Ubuntu 16.04 LTS 64bit

The docker images shows the following images:

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE
k8s.gcr.io/kube-apiserver-amd64            v1.11.2             821507941e9c        3 weeks ago         187 MB
k8s.gcr.io/kube-controller-manager-amd64   v1.11.2             38521457c799        3 weeks ago         155 MB
k8s.gcr.io/kube-proxy-amd64                v1.11.2             46a3cd725628        3 weeks ago         97.8 MB
k8s.gcr.io/kube-scheduler-amd64            v1.11.2             37a1403e6c1a        3 weeks ago         56.8 MB
k8s.gcr.io/coredns                         1.1.3               b3b94275d97c        3 months ago        45.6 MB
k8s.gcr.io/etcd-amd64                      3.2.18              b8df3b177be2        4 months ago        219 MB
k8s.gcr.io/pause                           3.1                 da86e6ba6ca1        8 months ago        742 kB

Full logs can be found here : https://pastebin.com/T5V0taE3

I didn't found any solution on internet.

EDIT:

docker ps -a output:

ubuntu@ubuntu-HP-Pavilion-15-Notebook-PC:~$ docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS

journalctl -xeu kubelet output:

journalctl -xeu kubelet
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Sep 01 10:40:05 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: Started kubelet: T
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-d
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-d
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: F0901 10:40:06.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: M
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: U
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: F
lines 788-810/810 (END)
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
Sep 01 10:40:05 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-driver has been deprecated, This parameter should be set via the
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: Flag --cgroup-driver has been deprecated, This parameter should be set via the
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.117131    9107 server.go:408] Version: v1.11.2
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.117406    9107 plugins.go:97] No cloud provider specified.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.121192    9107 certificate_store.go:131] Loading cert/key pair 
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: I0901 10:40:06.145720    9107 server.go:648] --cgroups-per-qos enabled, but --
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC kubelet[9107]: F0901 10:40:06.146074    9107 server.go:262] failed to run Kubelet: Running wi
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Unit entered failed state.
Sep 01 10:40:06 ubuntu-HP-Pavilion-15-Notebook-PC systemd[1]: kubelet.service: Failed with result 'exit-code'.
~


          PORTS               NAMES

Any help/suggestion/comment would be appreciated.

-- Akshay Sood
kubeadm
kubectl
kubernetes

Similar Questions

kubernetes does not track pods success count for job completion
kubernetes DNS lookup failure
Kubernetes request to service timeout value
helm chart "stable/mongodb-replicaset" is stuck when 2nd pods bootstraps / is added to replicaset
Error while trying to convert compose file to kubernetes manifest

5 Answers

2/29/2020

You should not use with swap even have been closed by swapoff -a.

You should disable in setting file /etc/fstab

for me:

root@kali:~# cat /etc/fstab

UUID=ce70d41a-0ce7-42bb-a318-d89369f93b28 / ext4 errors=remount-ro 0 1

#swap was on /dev/sda5 during installation

UUID=2271021b-2aed-4b49-9757-54e7d42ef33e none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

after editing:

root@kali:~# cat /etc/fstab

UUID=ce70d41a-0ce7-42bb-a318-d89369f93b28 / ext4 errors=remount-ro 0 1

#swap was on /dev/sda5 during installation

#UUID=2271021b-2aed-4b49-9757-54e7d42ef33e none swap sw 0 0

/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0

then.. reboot your machine

Note: Don't use reset after rebooting before kubeadm init via:

kubeadm reset

-- dose
Source: StackOverflow
11/5/2019

if you are running on openstack or any cloud please make sure to allow the ports in security groups TCP | Inbound | 6443* | Kubernetes API server

in my case this was the error

-- jobin at
Source: StackOverflow
6/15/2019

sudo sed -i '/ swap / s/^/#/' /etc/fstab

I spent last few days to solve the same problem and this also worked for me. Don't know whay it doesn't have same effect only with 'sudo swapoff -a'.

[kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get http://localhost:10248/healthz: dial tcp 127.0.0.1:10248: connect: connection refused.

And these error logs are really confusing because there is no clue there showing that this problem may be related to swap configuraiton.

-- late spring
Source: StackOverflow
9/6/2018

The error was fixed by

sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab

reboot the machine.

-- Akshay Sood
Source: StackOverflow
9/1/2018

As you use Kubernetes 1.11.2, it's useful to quote from CHANGELOG-1.11.md:

kubeadm now detects the Docker cgroup driver and starts the kubelet with the matching driver. This eliminates a common error experienced by new users in when the Docker cgroup driver is not the same as the one set for the kubelet due to different Linux distributions setting different cgroup drivers for Docker, making it hard to start the kubelet properly.

It seems to me that on your node some wrong parameter is passed to kubelet and this one doesn't start.

  1. First of all check that your docker has cfgroups with the command:

    docker info |grep -i cgroup

Output should be:

Cgroup Driver: cgroupfs

  1. Now go on your node and look for the kubelet service script, probably in /etc/systemd/system/kubelet.service (or similar name) and remove all parameters regarding cfgroup inside that script.

  2. Try to restart the kubelet service

  3. Check the kubelet's log again (journalctl -xeu kubelet)

-- Nicola Ben
Source: StackOverflow