I am trying to monitor filesystem usage for pods in k8s. I am using Kubernetes (microk8s) and hostpath persistent volumes. I am running Kafka along with a number of producers to see what happens when I go past the PVC size limit among other things. I have tried getting information from the API server but it is not reported there. Since it is only using hostpath, that kind of makes sense. It is not a dynamic volume system. Doing df on the host just shows all of the volumes with the same utilization as the root filesystem. This is the same result using exec -- df within the container. There are no pvcRefs on the containers using api server, which kind of explains why the dashboard doesn't have this information. Is this a dead end or does someone have a way around this limitation? I am now wondering if the PVC limits will be enforced.
Since with hostPath your data is stored directly on the worker you won't be able to monitor the usage. Using hostPath has many drawbacks and while its good for testing it should not be used for some prod system. Keeping the data directly on the node is dangerous and in the case of node failure/replacement you will loose it. Other disadvantages are:
Pods created from the same pod template may behave differently on different nodes because of different hostPath file/dir contents on those nodes
Files or directories created with HostPath on the host are only writable by root. Which means, you either need to run your container process as root or modify the file permissions on the host to be writable by non-root user, which may lead to security issues
hostPath volumes should not be used with Statefulsets.
As you already found out it would be good idea to move on from hostPath towards something else.