I'm trying to get disk usage metrics of PVs via Prometheus from the Kubernetes API. I receive a 403 forbidden message in Prometheus, but can access these metrics both locally and within the cluster using a JWT token.

I can receive disk capacity metrics but not actual usage.

I've tried changing the prometheus cluster role to include nodes/metrics and nodes/proxy as resources, and I've also tried to enable webhook authentication on the kubelet.

Either in K8s 1.15 and above they've completely removed the ability to see disk usage metrics, or I'm doing something completely wrong here. Are these metrics endpoints even usable anymore? (https://kubernetes.default.svc:443/api/v1/nodes/{node}/proxy/metrics)