I try to get Total and Free disk space on my Kubernetes VM so I can display % of taken space on it. I tried various metrics that included "filesystem" in name but none of these displayed correct total disk size. Which one should be used to do so?
Here is a list of metrics I tried
node_filesystem_size_bytes
node_filesystem_avail_bytes
node:node_filesystem_usage:
node:node_filesystem_avail:
node_filesystem_files
node_filesystem_files_free
node_filesystem_free_bytes
node_filesystem_readonly
According kubernetes-mixin you can use node:node_filesystem_usage, but if this metrics won't show your correct vales please ensure that device is set correctly.
I highly recommend using Prometheus Graph or Graphana Explore tab to check all available metrics.
According to my Grafana dashboard, the following metrics work nicely for alerting for available space,
100 - ((node_filesystem_avail_bytes{mountpoint="/",fstype!="rootfs"} * 100) / node_filesystem_size_bytes{mountpoint="/",fstype!="rootfs"})
The formula gives out the percentage of available space on the pointed disk. Make sure you include the mountpoint and fstype within the metrics.