How can I find out why a succesful kubernetes MountVolume step doesnt result in any Mounts in a docker container?

8/5/2017

I've defined a mount point like so:

      - name: dir-graphite
        configMap:
          name: hub-logstash-grafana
          items:
            - key: logstash.conf.file
              path: config

with a later volume declaration:

      volumes:
        volumeMounts:
        - mountPath: "/opt/blackduck/hub/logs"
          name: dir-webapp
        - mountPath: "/var/lib/logstash/data"
          name: dir-logstash
        - mountPath: "/tmp/x"
          name: dir-graphite

In kubernetes 1.6.6, I see

Aug 05 02:02:59 ip-10-0-26-84 kubelet[30344]: I0805 02:02:59.912640   30344 operation_generator.go:597] 
MountVolume.SetUp succeeded for volume "kubernetes.io/configmap/a47ebff8-7976-11e7-8369-12207729cdd2-dir-graphite" (spec.Name: "dir-graphite") pod "a47ebff8-7976-11e7-8369-12207729cdd2" (UID: "a47ebff8-7976-11e7-8369-12207729cdd2").

That is, I can see the mount set up operation succeeding for my config map, however, when i inspect the actual created container, I see no associated mount:

"Mounts": [
            {
                "Source": "/var/lib/kubelet/pods/a47ebff8-7976-11e7-8369-12207729cdd2/etc-hosts",
                "Destination": "/etc/hosts",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/kubelet/pods/a47ebff8-7976-11e7-8369-12207729cdd2/containers/logstash/86b079de",
                "Destination": "/dev/termination-log",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Source": "/var/lib/kubelet/pods/a47ebff8-7976-11e7-8369-12207729cdd2/volumes/kubernetes.io~secret/default-token-2t0cl",
                "Destination": "/var/run/secrets/kubernetes.io/serviceaccount",
                "Mode": "ro",
                "RW": false,
                "Propagation": "rprivate"
            }
-- jayunit100
kubernetes
volume

1 Answer

8/5/2017

The following snippet is indeed correct:

 31         containers:
 32         - image: blackducksoftware/hub-logstash:4.0.0
 33           name: logstash
 34           volumeMounts:
 35           - name: dir-graphite
 36             mountPath: /tmp/x

I beleive there is a situation where if you have a minor indentation off, in 1.6, validation isnt very strict, so you get a no-op where a volume is loaded but the mount paths don't necessarily get read into anything.

For the record , if the volume is truly attached, you should see this in your mount points when running docker inspect:

 "Mounts": [
            {
                "Source": "/var/lib/kubelet/pods/bf655bb7-7985-11e7-8369-12207729cdd2/volumes/kubernetes.io~configmap/dir-graphite",
                "Destination": "/tmp/x",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            },

Moral of the story: In the case of ConfigMaps (and likely any other volume that is lazily mounted by a kubelet), A successful MountVolume.SetUp log at the kubelet level doesn't gaurantee that your files were mounted into a container, rather, it only means that the kubelet was able to create a volume corresponding to one of your defined ConfigMap volumes.

-- jayunit100
Source: StackOverflow