I've run into some disk pressure issues under GKE (too many job pods that weren't cleaned up) and I want to somehow be alerted next time I run low on disk space. I looked in Stackdriver Monitoring, but I only see disk read/write bytes/operations. In order to get disk usage/utilization, I think I have to install the stackdriver agent, but I'm not sure if this is possible under GKE. Is there some way to monitor disk utilization for GKE nodes?
Was searching for this today. The recommended answer relates to a legacy metric. Now the following metric is available: metric.type="kubernetes.io/node/ephemeral_storage/allocatable_bytes" resource.type="k8s_node"
when installing the Stackdriver Agent, there are per default only these two disk metrics available: container/disk/bytes_total and container/disk/bytes_used. in fact, one cannot really monitor the disk utilization alike that.
rather see Creating Custom Metrics, with an Alerting Policy.