Many occurrences of unmount in kubelet

5/17/2016

In kubernetes 1.2 every 20 seconds in our kubelet logs we see many of the following log lines.

Is this expected behaviour? They seem to coincide with high CPU usage from the kubelet process. On this particular node no new pods have been deployed for 2 days - so this does not coincide with the creation of pods.

May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706297    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 00126ffb-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706464    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 001285f0-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706502    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 001285f0-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706617    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 00129876-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706655    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 00129876-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706777    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 0012c849-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706827    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 0012c849-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706952    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 004cfabd-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.706991    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 004cfabd-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707122    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 0127085b-16cb-11e6-9cb4-00163e1b3b81
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707161    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 0127085b-16cb-11e6-9cb4-00163e1b3b81/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707294    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 01b0b20c-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707340    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 01b0b20c-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707453    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 01c01c42-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707490    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 01c01c42-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707607    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 0233ff42-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707644    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 0233ff42-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707784    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 027dee6c-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707821    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 027dee6c-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.707950    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 03e74e54-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708018    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 03e74e54-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708161    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 0e54291f-18f2-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708207    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 0e54291f-18f2-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708316    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 14f313d5-16cd-11e6-9cb4-00163e1b3b81
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708355    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 14f313d5-16cd-11e6-9cb4-00163e1b3b81/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708471    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 2bf78c9d-1904-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708509    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 2bf78c9d-1904-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708634    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 3724f295-16c7-11e6-9cb4-00163e1b3b81
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708673    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 3724f295-16c7-11e6-9cb4-00163e1b3b81/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708792    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 383a146a-17a2-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708831    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 383a146a-17a2-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.708959    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 42e0108a-16c5-11e6-9cb4-00163e1b3b81
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709025    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 42e0108a-16c5-11e6-9cb4-00163e1b3b81/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709151    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 4c745286-179b-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709189    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 4c745286-179b-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709301    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 56d21cb3-185a-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709339    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 56d21cb3-185a-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709451    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 5a150a98-185a-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709490    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 5a150a98-185a-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709613    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 5d8ee492-185a-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709674    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 5d8ee492-185a-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709787    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 6ad52938-1834-11e6-874e-00163e1b3b80
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709836    6805 volumes.go:316] Used volume plugin "kubernetes.io/secret" to unmount 6ad52938-1834-11e6-874e-00163e1b3b80/kubernetes.io~secret
May 17 10:54:16 boxtest-2 kubelet[6805]: I0517 10:54:16.709953    6805 volumes.go:234] Making a volume.Cleaner for volume kubernetes.io~secret/default-token-ckjjp of pod 6bf10079-1903-11e6-874e-00163e1b3b80
-- Andy Smith
kubernetes

1 Answer

5/17/2016

Kubelet periodically checks the volumes and examines whether any of them needs to be unmounted/detached. In the process of doing that, it gets a list of volumes and their unmounter/detachers, in case cleanup is required. These messages are a side-effect of this checkup, and they do not mean that the volumes are unmounted. The verbosity of the log messages have been adjusted in HEAD (post v1.2+) to avoid further confusion.

-- Yu-Ju Hong
Source: StackOverflow