This question is regarding kubernetes storage. I am using a local kubernetes cluster where while some applications to be deployed need to be backed by pvcs. The PVC are provisioned dynamically . However, sometimes when there is no storage left on the cluster the pvc request just gets stuck in forever pending state.
Is there any way that the available storage on the kubernetes cluster be checked? Checked extensively in the docs and it is just not clear how to check remaining storage capacity on a kubernetes cluster.
Also, as per kubernetes docs the capacity of a node is different and the pvc allocation is bound to the pv which are a completely separate cluster resource just like nodes.
In that case what storage do I need to check to find if there's any space available for say an x gb dynamic pvc? Also, how do i check it?
You can use tools for monitoring resources.
One of it is Prometheus you can combine it with Grafana to visualize collected metrics.
Also take a look on compute-resources-consumption-monitoring.
When local ephemeral storage is used, it is monitored on an ongoing basis by the kubelet. The monitoring is performed by scanning each emptyDir volume, log directories, and writable layers on a periodic basis. Starting with Kubernetes 1.15, emptyDir volumes (but not log directories or writable layers) may, at the cluster operator’s option, be managed by use of project quotas. Project quotas were originally implemented in XFS, and have more recently been ported to ext4fs. Project quotas can be used for both monitoring and enforcement; as of Kubernetes 1.16, they are available as alpha functionality for monitoring only.
Quotas are faster and more accurate than directory scanning. When a directory is assigned to a project, all files created under a directory are created in that project, and the kernel merely has to keep track of how many blocks are in use by files in that project. If a file is created and deleted, but with an open file descriptor, it continues to consume space. This space will be tracked by the quota, but will not be seen by a directory scan.
To enable use of project quotas, the cluster operator must do the following:
LocalStorageCapacityIsolationFSQuotaMonitoring=true feature gate in the kubelet configuration. This defaults to false in Kubernetes 1.16, so must be explicitly set to trueMake sure that the root partition (or optional runtime partition) is mounted with project quotas enabled.