After starting kubernetes from this guide: http://kubernetes.io/docs/getting-started-guides/docker/ I get a lot of unused mount points on my node. It seems to depend on how many pods are running. Just now I had to unmount over 2600 mount points. When these build up it causes findmnt to take a lot of resources to run. The mount entries look like this:

tmpfs on /var/lib/kubelet/pods/599d6157-081e-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha type tmpfs (rw)

Does anybody know why these are not getting unmounted automatically? From the tutorial it seems to anticipate that you will have to clean some of these up (look under the Turning down your cluster section), but this seems excessive. A few days ago I had to clean up 22,000 or so of them because I had a mongo cluster and redis running for a while.

--- UPDATE --- After purging my system of the unused mounts, and waiting a few minutes, findmnt produces entries like this:

├─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha             tmpfs                           tmpfs               rw,relatime
│ └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha           tmpfs                           tmpfs               rw,relatime
│   └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha         tmpfs                           tmpfs               rw,relatime
│     └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha       tmpfs                           tmpfs               rw,relatime
│       └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha     tmpfs                           tmpfs               rw,relatime
│         └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha   tmpfs                           tmpfs               rw,relatime
│           └─/var/lib/kubelet/pods/02929977-0812-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs                           tmpfs               rw,relatime
├─/var/lib/docker/containers/c84ad9b0f2ec580bedef394aa46bb147ed6c4f1e9454cd3729459d9127c0986e/shm                         shm                             tmpfs               rw,nosuid,nodev,noe
├─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia             tmpfs                           tmpfs               rw,relatime
│ └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia           tmpfs                           tmpfs               rw,relatime
│   └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia         tmpfs                           tmpfs               rw,relatime
│     └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia       tmpfs                           tmpfs               rw,relatime
│       └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia     tmpfs                           tmpfs               rw,relatime
│         └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia   tmpfs                           tmpfs               rw,relatime
│           └─/var/lib/kubelet/pods/0eb8631e-0810-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-2vjia tmpfs                           tmpfs               rw,relatime
├─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha             tmpfs                           tmpfs               rw,relatime
│ └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha           tmpfs                           tmpfs               rw,relatime
│   └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha         tmpfs                           tmpfs               rw,relatime
│     └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha       tmpfs                           tmpfs               rw,relatime
│       └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha     tmpfs                           tmpfs               rw,relatime
│         └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha   tmpfs                           tmpfs               rw,relatime
│           └─/var/lib/kubelet/pods/fae71387-08aa-11e6-a512-0090f5ea551f/volumes/kubernetes.io~secret/default-token-kkzha tmpfs                           tmpfs               rw,relatime
├─/var/lib/docker/containers/5392d49f5140274ddcfbe757cf6a07336aa60975f3ea122d865a3b80f5540c1f/shm  

-- Update #2 -- This is how I am starting kubelet

ARCH=amd64
DNS_IP=10.0.0.10
K8S_VERSION=$(curl -sS https://storage.googleapis.com/kubernetes-release/release/stable.txt)


docker run \
    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --net=host \
    --pid=host \
    --privileged=true \
    --name=kubelet \
    -d \
    gcr.io/google_containers/hyperkube-${ARCH}:${K8S_VERSION} \
    /hyperkube kubelet \
        --containerized \
        --hostname-override="127.0.0.1" \
        --address="0.0.0.0" \
        --api-servers=http://localhost:8080 \
        --config=/etc/kubernetes/manifests \
        --cluster-dns=$DNS_IP \
        --cluster-domain=cluster.local \
        --allow-privileged=true --v=2

In looking at some other suggestions (thanks Thibault Deheurles), I tried removing --containerized and --volume=/:/rootfs:ro, but that caused k8s to not start at all.

-- UPDATE #3 -- I tried adding the mount flag ,shared to my /var/lib/kubelet volume command, it now looks like this:

--volume=/var/lib/kubelet/:/var/lib/kubelet:rw,shared

This didn't make a difference.

However, I noticed while tailing my kubelet docker container's logs, that this message occurs every time I get a new mount...

2016-04-26T20:30:52.447842722Z I0426 20:30:52.447559   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1

Failed GC comment also appears in log... Here are a few more entries

2016-04-26T20:38:11.436858475Z E0426 20:38:11.436757   21740 kubelet.go:956] Image garbage collection failed: non-existent label "docker-images"
2016-04-26T20:38:12.448049454Z I0426 20:38:12.447852   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:38:52.448175137Z I0426 20:38:52.447949   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:39:14.447892769Z I0426 20:39:14.447649   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:39:28.441137221Z I0426 20:39:28.440920   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:40:20.441118739Z I0426 20:40:20.441018   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:40:22.447832573Z I0426 20:40:22.447590   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:40:53.447612605Z I0426 20:40:53.447534   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1
2016-04-26T20:41:27.449053007Z I0426 20:41:27.448820   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/d95a6048198f747c5fcb74ee23f1f25c/volumes/kubernetes.io~empty-dir/data: exit status 1
2016-04-26T20:41:30.440974280Z I0426 20:41:30.440889   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/6bc8072c-0be9-11e6-b3e6-0090f5ea551f/volumes/kubernetes.io~empty-dir/etcd-storage: exit status 1
2016-04-26T20:41:58.441001603Z I0426 20:41:58.440906   21740 nsenter_mount.go:185] Failed findmnt command for path /var/lib/kubelet/pods/1df6a8b4d6e129d5ed8840e370203c11/volumes/kubernetes.io~empty-dir/varetcd: exit status 1

-- UPDATE #4 -- @PaulMorie asked about the mount/findmnt versions

$ which findmnt
/bin/findmnt

$ uname -a
Linux andromeda 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.4 LTS
Release:    14.04
Codename:   trusty

$ dpkg -L mount | grep findmn
/usr/share/man/man8/findmnt.8.gz
/bin/findmnt

$ dpkg -l mount
ii  mount                                2.20.1-5.1ubuntu20.7    amd64                   Tools for mounting and manipulating filesystems

-- UPDATE #5 -- @tsaarni asked what I did to solve this problem... Here is my hack

[eric@andromeda [feature/k8s-packaging-openvpn]util]$ cat clean-mounts.sh 
#!/bin/bash
counter=0
for  m in $( mount| grep secret | awk '{print $3}' ); do 
    sudo umount $m
    counter=$[counter + 1]
done
echo "cleaned $counter mounts"


[eric@andromeda [feature/k8s-packaging-openvpn]util]$ cat clean-mounts-watcher.sh 
#!/bin/bash
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd)"
while : ; do $DIR/clean-mounts.sh ; sleep 60; done