I have a K8s 1.7 cluster using vSphere as persistent storage provider. I have also deployed Prometheus, node_exporter and kube-state-metrics.
I'm trying to find a way to monitor a persistent volume's usage using Prometheus. I have added custom labels to some PVs, eg. app=rabbitmq-0, etc.
How can I combine kube_persistentvolume_labels with node_filesystem_size metrics so that I can query PV usage using my custom label?
PS.
I know that K8s 1.8 directly exposes these metrics from kubelet as mentioned in How to monitor disk usage of kubernetes persistent volumes? but currently a cluster upgrade is not an option.
Starting from (v1.3.0-rc.0 / 2018-03-23) in the kube-state-metrics, two metrics that can convert PersistentVolume and PersistenVolumeClaims labels to Prometheus labels accordingly were added:
kube_persistentvolume_labels
kube_persistentvolumeclaim_lables
To get more details about implementing aggregation of metrics based on labels, consider reading these articles: