Edit this file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf change systemd to cgroupfs". then restart the kubelet systemctl restart kubelet.

Possible cause

kubelet 1.7.3 not reading config file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf #50748

Solution

Troubleshooting kubeadm

If you are using CentOS and encounter difficulty while setting up the master node， verify that your Docker cgroup driver matches the kubelet config:

docker info | grep -i cgroup
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

If the Docker cgroup driver and the kubelet config don’t match, change the kubelet config to match the Docker cgroup driver. The flag you need to change is --cgroup-driver. If it’s already set, you can update like so:

sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

This can be replaced with:

CG=$(sudo docker info 2>/dev/null | sed -n 's/Cgroup Driver: \(.*\)/\1/p')
sed -i "s/cgroup-driver=systemd/cgroup-driver=$CG/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

To get minicube running on a CentOS 7, I need to start it with a --extra-config=kubelet.cgroup-driver=systemd as suggested in https://github.com/kubernetes/minikube/issues/2192.

On my environment it only worked the other way around. Setting systemd results always in an error. Here is my current setup

OS: CentOS 7.6.1810 
Minikube Version v1.0.0
Docker Version  18.06.2-ce

The solution for me was: Check /etc/docker/daemon.json and change systemd to cgroupfs

{
  "exec-opts": ["native.cgroupdriver=cgroupfs"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}

Then reload systemctl systemctl daemon-reload Kill the previous minikub config minikube delete and start the minikube again minikube start --vm-driver=none

Now check the command line the output should find cgroupfs in both outputs

docker info | grep -i cgroup
cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

In the end you should see

   kubectl is now configured to use "minikube"
=   Done! Thank you for using minikube!

Simple solution: Start your minikube with the Extra onfig parameter

--extra-config=kubelet.cgroup-driver=systemd

The complete command to start up minikube is the next line

minikube start --vm-driver=none --extra-config=kubelet.cgroup-driver=systemd

All the best and have fun

It looks like kubelet process did not load the right settings from the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf telling from the error message.

After getting more info from the chat, I think there are several possible ways to go:

1. Switch both kubelet and docker cgroup driver to cgroupfs Download docker from the repo below which uses cgroupfs by default.

   [dockerrepo] 
   name=Docker Repository 
   baseurl=https://yum.dockerproject.org/repo/main/centos/7 
   enabled=1 
   gpgcheck=1 
   gpgkey=https://yum.dockerproject.org/gpg

   And change the cgroup driver in kubelet conf as well. Check whether the error happens again and what kubelet loads from its conf.

2. Add more logs in kubelet code to debug it

   This is the logic kubelet uses to get conf from both sides

This is caused by miscofiguration during the initial startup. For example forgeting to change docker cgroup driver before executing kubeadm init command.

To remedy this under CentOS, open /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf or locate the file under your operating system. Locate the entry with EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env. Open this file and change the value of --cgroup-driver to systemd or to be the same as docker cgroup driver. Old Content:

KUBELET_KUBEADM_ARGS=--cgroup-driver=cgrouopfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

New Content:

KUBELET_KUBEADM_ARGS=--cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1

It may be better to do the reverse and make kubelet to use systemd

In Kubernetes site, they recommend using systemd https://kubernetes.io/docs/setup/production-environment/container-runtimes/ More details here

And you can change kubelet to use systemd by following https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/ adding this to /etc/sysconfig/kubelet

cat /etc/sysconfig/kubelet 
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd