TL;DR

Bind mounts are not working in kubernetes pods using a dind image. Ex.

mkdir code && \
docker run --rm --mount type=bind,source=$PWD/code/,target=/code/ alpine:latest sh -c 'echo "this" > code/test && ls -al code/ && cat code/test' && \
cat code/test

Background

I have a jenkins setup in k8s using stable helm chart. This spins a jenkins slave which provides a relatively safe way to use Docker-in-Docker (dind). We use this setup to spin up docker-compose for running our tests cases. The idea is that the same docker-compose setup that devs use to perform tests in their local hosts should be used by CI engine to perform tests. The test cases involve npx mocha tests which splits out test result in a path. If this path is bind mounted from the pod's file-system then the test result file created in the containers spun up by docker-compose should exist in the pods as well. However, while this behaviour is seen in localhost (Mac/ubuntu) it is not so in jenkins slaves as k8s pods.

Issue

run this command in localhost:

$ mkdir
$ docker run --rm --mount type=bind,source=$PWD/code/,target=/code/ alpine:latest sh -c 'echo "this" > code/test && ls -al code/ && cat code/test'
## output: 
## total 8
## drwxr-xr-x    3 root     root            96 May 29 06:59 .
## drwxr-xr-x    1 root     root          4096 May 29 06:59 ..
## -rw-r--r--    1 root     root             5 May 29 06:59 test
## this
$ cat code/test
## output: this

the same command in k8s jenkins slave:

 docker run --rm --mount type=bind,source=$PWD/code/,target=/code/ alpine:latest sh -c 'echo "this" > code/test && ls -al code/ && cat code/test'
## total 4
## drwxr-xr-x    2 root     root            18 May 29 05:45 .
## drwxr-xr-x    1 root     root            18 May 29 05:52 ..
## -rw-r--r--    1 root     root             5 May 29 05:52 test
## this
bash-4.4# ls -al code/
## total 0
## drwxr-xr-x    2 root     root             6 May 29 05:44 .
## drwxr-xr-x    1 root     root            74 May 29 05:44 ..

NOTE: I have deliberately run everything as root ins jenkin's slave pod to get around the issue of user-permissions for the purpose of this demo

What I have already tried

• using docker --volume instead of mount
• using all possible options for bind-propagation ( all though none but private works in mac - k8s' jenkins surprisingly does allow all possible values for bind-propagation - possible because it is on version 18.09.9-ce

More info

• dind image has been enabled in k8s by:
  • mounting /var/run/docker.sock
  • pod is not privileged
• version
  • localhost (MAC)

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

• k8s

docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:20:43 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.09.9-ce
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       039a7df
  Built:            Fri Nov  1 19:28:24 2019
  OS/Arch:          linux/amd64
  Experimental:     false

• docker system info storage driver details
  • localhost (MAC)

 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true

• k8s

Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true

• docker inspect
  • localhost (MAC)

[
    {
        "Id": "275342d559db6719bdc3b5b80c44ce961bb4583f727eabff5ccfdc2a5877d54b",
        "Created": "2020-05-29T06:04:36.333043942Z",
        "Path": "sh",
        "Args": [
            "-c",
            "echo \"this\" > code/test && ls -al code/ && cat code/test && sleep 10",
            "--name",
            "test123"
        ],
        "State": {
            "Status": "exited",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-05-29T06:04:36.58940417Z",
            "FinishedAt": "2020-05-29T06:04:46.616157405Z"
        },
        "Image": "sha256:f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a",
        "ResolvConfPath": "/var/lib/docker/containers/275342d559db6719bdc3b5b80c44ce961bb4583f727eabff5ccfdc2a5877d54b/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/275342d559db6719bdc3b5b80c44ce961bb4583f727eabff5ccfdc2a5877d54b/hostname",
        "HostsPath": "/var/lib/docker/containers/275342d559db6719bdc3b5b80c44ce961bb4583f727eabff5ccfdc2a5877d54b/hosts",
        "LogPath": "/var/lib/docker/containers/275342d559db6719bdc3b5b80c44ce961bb4583f727eabff5ccfdc2a5877d54b/275342d559db6719bdc3b5b80c44ce961bb4583f727eabff5ccfdc2a5877d54b-json.log",
        "Name": "/admiring_albattani",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Capabilities": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/Volumes/NonOS/ops/code/",
                    "Target": "/code/"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/2f47fcb09bc0395452eaeb3b26d917cd0cf41e3470b59a895708ea6698d6487a-init/diff:/var/lib/docker/overlay2/e78c363c622f70ae20f3a4bcfa55e442c9abc2d696330a1db22de60b7276bf43/diff",
                "MergedDir": "/var/lib/docker/overlay2/2f47fcb09bc0395452eaeb3b26d917cd0cf41e3470b59a895708ea6698d6487a/merged",
                "UpperDir": "/var/lib/docker/overlay2/2f47fcb09bc0395452eaeb3b26d917cd0cf41e3470b59a895708ea6698d6487a/diff",
                "WorkDir": "/var/lib/docker/overlay2/2f47fcb09bc0395452eaeb3b26d917cd0cf41e3470b59a895708ea6698d6487a/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/Volumes/NonOS/ops/code",
                "Destination": "/code",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "275342d559db",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "sh",
                "-c",
                "echo \"this\" > code/test && ls -al code/ && cat code/test && sleep 10",
                "--name",
                "test123"
            ],
            "Image": "alpine:latest",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "2dbae65e10eec270564e5952d41b843a10f277d477a24eaf2c0b5585923a0bd4",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/2dbae65e10ee",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "2375a159bfc5e9dce5d263c322c7b2827d79cefa88197f24a051603b29c89fc8",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

• k8s

[
    {
        "Id": "d3f6a13ed635ce9199945c9c54fbb2f65b190b46190822fb781eb70ce0c0971a",
        "Created": "2020-05-29T06:03:52.837737029Z",
        "Path": "sh",
        "Args": [
            "-c",
            "echo \"this\" > code/test && ls -al code/ && cat code/test && sleep 10",
            "--name",
            "test123"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 30976,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2020-05-29T06:03:53.263122903Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a",
        "ResolvConfPath": "/var/lib/docker/containers/d3f6a13ed635ce9199945c9c54fbb2f65b190b46190822fb781eb70ce0c0971a/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/d3f6a13ed635ce9199945c9c54fbb2f65b190b46190822fb781eb70ce0c0971a/hostname",
        "HostsPath": "/var/lib/docker/containers/d3f6a13ed635ce9199945c9c54fbb2f65b190b46190822fb781eb70ce0c0971a/hosts",
        "LogPath": "/var/lib/docker/containers/d3f6a13ed635ce9199945c9c54fbb2f65b190b46190822fb781eb70ce0c0971a/d3f6a13ed635ce9199945c9c54fbb2f65b190b46190822fb781eb70ce0c0971a-json.log",
        "Name": "/musing_stallman",
        "RestartCount": 0,
        "Driver": "overlay2",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {
                    "max-file": "10",
                    "max-size": "10m"
                }
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "no",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": [],
            "DnsOptions": [],
            "DnsSearch": [],
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "shareable",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": [],
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": [],
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "//code/",
                    "Target": "/code/"
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/d2f3e1dc5a3a6531903ce305f7c26def037066cabf6e785fe3eb5541c7cc7266-init/diff:/var/lib/docker/overlay2/0514324ae287d8aeb6f7a7866eff5c3414e35c761fd3ce3daf5948a080698ba3/diff",
                "MergedDir": "/var/lib/docker/overlay2/d2f3e1dc5a3a6531903ce305f7c26def037066cabf6e785fe3eb5541c7cc7266/merged",
                "UpperDir": "/var/lib/docker/overlay2/d2f3e1dc5a3a6531903ce305f7c26def037066cabf6e785fe3eb5541c7cc7266/diff",
                "WorkDir": "/var/lib/docker/overlay2/d2f3e1dc5a3a6531903ce305f7c26def037066cabf6e785fe3eb5541c7cc7266/work"
            },
            "Name": "overlay2"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/code",
                "Destination": "/code",
                "Mode": "",
                "RW": true,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "d3f6a13ed635",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "sh",
                "-c",
                "echo \"this\" > code/test && ls -al code/ && cat code/test && sleep 10",
                "--name",
                "test123"
            ],
            "Image": "alpine:latest",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "NetworkDisabled": true,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "2d1bbff83058acd00b1fa98d8614628942686cf5eb9ae866fbf37466ee456564",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/2d1bbff83058",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]