Failed to use hostPath '/var/lib/docker/containers' as volume in Kubernetes

9/11/2018

I have failed to use HostPath /var/lib/docker/containers as a volume with the following error:

 Error response from daemon: linux mounts: Path /var/lib/docker/containers is 
 mounted on /var/lib/docker/containers but it is not a shared or slave mount.

Here is my YAML spec (note: this is just an example for reproducing my problem in doing log collection):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
  namespace: logging
  labels:
    app: test
spec:
  selector:
    matchLabels:
      app : test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - name: nginx
          image: nginx:stable-alpine
          securityContext:
            privileged: true
          ports:
          - containerPort : 8003
      volumeMounts:
      - name: docker
        mountPath: /var/lib/docker/containers
        readOnly: true
  volumes:
    - name: docker
      hostPath:
        path: /var/lib/docker/containers

And my kubernetes version.

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.1", 
GitCommit:"d4ab47518836c750f9949b9e0d387f20fb92260b", GitTreeState:"clean", 
BuildDate:"2018-04-12T14:26:04Z", GoVersion:"go1.9.3", Compiler:"gc", 
Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", 
GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", 
BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", 
Platform:"linux/amd64"}

Very much appreciating your help!

-- yero
docker
kubernetes

1 Answer

9/11/2018

very appreciated for any help!

You are most probably hit by a version specific issue:

/var/lib/docker/containers is intentionally mounted by Docker with private mount
propagation and thus conflicts with Kubernetes trying to mount this directory
as rslave when running the container

You should try with 1.10.3+ where it is resolved. See the official changelog for kubernetes and check entry related to "Default mount propagation". Also check related (see the error) fluentd issue for more in-depth analysis.

Now, with that said...

David's seasoned comment with question and caution word still stands and I second that: this is quite an eyebrow raiser - nginx pod digging deep into docker engine internals (hope it is just for sake of minimal reproducible example, or log collection case, you know, something...)... Just make sure you know exactly what you are doing and why.

-- Const
Source: StackOverflow