docker start failed after adding daemon.json file

1/7/2020

I'm trying to install Kubernetes on CentOS 7.7, therefore, I have to install docker first. I followed Kubernetes Documentation to install docker-ce and modify daemon.json file.

$ yum install yum-utils device-mapper-persistent-data lvm2
$ yum-config-manager --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo
$ yum update && yum install \
    containerd.io-1.2.10 \
    docker-ce-19.03.4 \
    docker-ce-cli-19.03.4
$ mkdir /etc/docker
$ cat > /etc/docker/daemon.json <<EOF
  {
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
      "max-size": "100m"
    },
    "storage-driver": "overlay2",
    "storage-opts": [
      "overlay2.override_kernel_check=true"
    ]
  }
  EOF

$ mkdir -p /etc/systemd/system/docker.service.d
$ systemctl daemon-reload
$ systemctl start docker

When started docker service, it said:

Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
$ systemctl status -l docker.service
docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2020-01-07 14:44:11 UTC; 7min ago
     Docs: https://docs.docker.com
  Process: 9879 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
 Main PID: 9879 (code=exited, status=1/FAILURE)

Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 14:44:09 love61y2222c.mylabserver.com systemd[1]: docker.service failed.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: docker.service holdoff time over, scheduling restart.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Stopped Docker Application Container Engine.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: start request repeated too quickly for docker.service
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 14:44:11 love61y2222c.mylabserver.com systemd[1]: docker.service failed.
$ journalctl -xe
.
.
-- Unit docker.service has begun starting up.
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.722780008Z" level=info msg="Starting up"
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728447514Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728479813Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728510943Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.728526075Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.732325726Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733844225Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733880664Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.733898044Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: time="2020-01-07T15:28:25.743421350Z" level=warning msg="Using pre-4.0.0 kernel for overlay2, mount failures may require
Jan 07 15:28:25 love61y2223c.mylabserver.com dockerd[29628]: failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: Unit docker.service entered failed state.
Jan 07 15:28:25 love61y2223c.mylabserver.com systemd[1]: docker.service failed.

Could anyone tell me why docker service start failed after modifying daemon.json file? And how to specify cgroupdriver, default log-driver and default storage-driver in the right way?

Any suggestion will be greatly appreciated. Thanks.

-- tan
docker
install
kubernetes

1 Answer

1/7/2020

This error is pointing to an issue forcing docker to use overlay2 without the proper backing filesystem:

failed to start daemon: error initializing graphdriver: overlay2: the backing xfs filesystem is formatted without d_type

See docker's table for details on backing filesystem requirements for the different storage drivers: https://docs.docker.com/storage/storagedriver/#supported-backing-filesystems

The fix is to remove the storage driver settings, or fix the backing filesystem with the needed options to support overlay2:

  {
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
      "max-size": "100m"
    }
  }

For details on changing the xfs options, that appears to require rebuilding the filesystem. See this answer for more details on the needed steps.

-- BMitch
Source: StackOverflow