Use case

I am operating a kafka cluster in Kubernetes which is heavily dependent on a a proper disk performance (IOPS, throughput etc.). I am using Google's compute engine disks + Google kubernetes engine. Thus I know that the disks I created have the following approx limits:

IOPS (Read/Write): 375 / 750 Throughput in MB/s (Read/Write): 60 / 60

The problem

Even though I know the approx IOPS and throughput limits I have no idea what I am actually using at the moment. I'd like to monitor it with prometheus + grafana but I couldn't find anything which would export disk io stats for persistent volumes. The best I found were disk space stats from kubelet:

kubelet_volume_stats_capacity_bytes
kubelet_volume_stats_available_bytes

The question

What possibilities do I have to monitor (preferably via prometheus) the disk io usage for my kafka persistent volumes attached in Kubernetes?

Edit:

Another find I made is using node-exporter's node_disk_io metric:

rate(node_disk_io_time_seconds_total[5m]) * 100

Unfortunately the result doesn't contain a nodename, or even a persistent volume (claim) name. Instead it has device (e.g. 'sdb') and an instance ( e.g. '10.90.206.10') label which are the only labels which would somehow allow me to monitor a specific persistent volume. The downside of these labels are that they are dynamic and can change with a pod restart or similiar.