How to map the guids under /var/lib/kubelet/pods to the actual pods

11/7/2021

I was trying to debug some mount problems and the mount logs led me to paths under /var/lib/kubelet/pods, i.e

/var/lib/kubelet/pods/f6affad1-941d-4df1-a0b7-38e3f2ab99d5/volumes/kubernetes.io~nfs/my-pv-e0dbe341a6fe475c9029fb372e

How can I map the guid of the root directory under pods to the actual running pod or container?

(f6affad1-941d-4df1-a0b7-38e3f2ab99d5 in the example above)

I don't see any correlation to the values returned by kubectl or crictl.

-- Mugen
kubectl
kubernetes
kubernetes-pod

1 Answer

11/7/2021

They're the .metadata.uid of the Pod; one can map them back by using your favorite mechanism for querying all pods and filtering on its .metadata.uid, and optionally restricting to just those pods scheduled on that Node if you have a so many Pods as to make the -A infeasible

for d in /var/lib/kubelet/pods/*; do
  p_u=$(basename "$d")
  kubectl get po -A -o json | \
    jq --arg pod_uuid "$p_u" -r '.items[] 
      | select(.metadata.uid == $pod_uuid) 
      | "uuid \($pod_uuid) is \(.metadata.name)"'
done

I'm sure there is a -o jsonpath= or -o gotemplate= form that removes the need for jq but that'd be a lot more work to type out in a textarea

with regard to your crictl question, I don't this second have access to my containerd cluster, but the docker based one labels the local containers with io.kubernetes.pod.uid so I would guess containerd does something similar:

            "Labels": {
                "annotation.io.kubernetes.container.hash": "e44bee94",
                "annotation.io.kubernetes.container.restartCount": "4",
                "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
                "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
                "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
                "io.kubernetes.container.logpath": "/var/log/pods/kube-system_storage-provisioner_b4aa3b1c-62c1-4661-a302-4c06b305b7c0/storage-provisioner/4.log",
                "io.kubernetes.container.name": "storage-provisioner",
                "io.kubernetes.docker.type": "container",
                "io.kubernetes.pod.name": "storage-provisioner",
                "io.kubernetes.pod.namespace": "kube-system",
                "io.kubernetes.pod.uid": "b4aa3b1c-62c1-4661-a302-4c06b305b7c0",
                "io.kubernetes.sandbox.id": "3950ec60121fd13116230cad388a4c6c4e417c660b7da475436f9ad5c9cf6738"
            }
-- mdaniel
Source: StackOverflow