I'm running a simple RWX NFS server in my Kubernetes cluster on GKE, and have installed the Prometheus Operator Helm chart. Prometheus pulls a lot of data from the various metrics sources in the k8s cluster, including kubelet. Unfortunately, it doesn't return a value for kubelet_volume_stats_available_bytes for all the available PVC's in the cluster.

When I look in Prometheus, it's reporting the stats for the PV/PVC's setup by Kafka, which are gce-pd instances:

$ ssh gke-prometheus-dev-k8s-standard-0e601273-tklt
gke-prometheus-dev-k8s-standard-0e601273-tklt ~ $ curl -SsL localhost:10255/metrics | grep kubelet_volume_stats_available_bytes
# HELP kubelet_volume_stats_available_bytes Number of available bytes in the volume
# TYPE kubelet_volume_stats_available_bytes gauge
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="datadir-kafka-0"} 1.042683904e+10
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="datadir-kafka-1"} 1.04267776e+10
kubelet_volume_stats_available_bytes{namespace="default",persistentvolumeclaim="datadir-kafka-2"} 1.0426851328e+10

$ ssh gke-prometheus-dev-k8s-standard-0e601273-0tzf
gke-prometheus-dev-k8s-standard-0e601273-0tzf ~ $ curl -SsL localhost:10255/metrics | grep kubelet_volume_stats_available_bytes
$

Connecting to the different nodes in the cluster, and querying the kubelet metrics-server shows the above responses, which match up with what Prometheus is showing me.

Querying the cluster with kubectl returns the following:

$ kubectl get pv,pvc
NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                     STORAGECLASS   REASON   AGE
persistentvolume/nfs-file-store                             10Gi       RWX            Retain           Bound    default/nfs-file-store                            13m
persistentvolume/nfs-logs                                   10Gi       RWX            Retain           Bound    default/nfs-logs                                  13m
persistentvolume/pvc-93464537-0f1c-448a-841c-18ce5645e076   10Gi       RWO            Delete           Bound    default/datadir-kafka-2   standard                4d20h
persistentvolume/pvc-bbbece3f-cafe-4fed-82cb-255b85e79c2d   10Gi       RWO            Delete           Bound    default/datadir-kafka-0   standard                4d21h
persistentvolume/pvc-f0ce3653-42b3-4fce-9b55-121505ddceb5   10Gi       RWO            Delete           Bound    default/datadir-kafka-1   standard                4d20h

NAME                                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
persistentvolumeclaim/datadir-kafka-0   Bound    pvc-bbbece3f-cafe-4fed-82cb-255b85e79c2d   10Gi       RWO            standard       4d21h
persistentvolumeclaim/datadir-kafka-1   Bound    pvc-f0ce3653-42b3-4fce-9b55-121505ddceb5   10Gi       RWO            standard       4d20h
persistentvolumeclaim/datadir-kafka-2   Bound    pvc-93464537-0f1c-448a-841c-18ce5645e076   10Gi       RWO            standard       4d20h
persistentvolumeclaim/nfs-file-store    Bound    nfs-file-store                             10Gi       RWX                           13m
persistentvolumeclaim/nfs-logs          Bound    nfs-logs                                   10Gi       RWX                           13m

Any pointers as to why kubelet isn't seeing these disks?