Does anyone know how to get a sorted list of pods on a given node based on their disk size consumption?
The below command helps me in listing pods based on a given node but my requirement is to list those pods which are causing high disk usage as part of an investigation process for solving the DiskPressure eviction state.
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<NODE_NAME>I was able to find commands to list cpu and memory usage of nodes (ref: https://github.com/kubernetes/kubernetes/issues/17512#issuecomment-317757388) but nothing related to node disk usage.
alias util='kubectl get nodes --no-headers | awk '\''{print $1}'\'' | xargs -I {} sh -c '\''echo {} ; kubectl describe node {} | grep Allocated -A 5 | grep -ve Event -ve Allocated -ve percent -ve -- ; echo '\'''
# Get CPU request total (we x20 because because each m3.large has 2 vcpus (2000m) )
alias cpualloc='util | grep % | awk '\''{print $1}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*20), "%\n" } }'\'''
# Get mem request total (we x75 because because each m3.large has 7.5G ram )
alias memalloc='util | grep % | awk '\''{print $5}'\'' | awk '\''{ sum += $1 } END { if (NR > 0) { print sum/(NR*75), "%\n" } }'\'''Unfortunately you won't be able to achieve this with kubectl as it does not have this kind of functionality. One way to go is login to node and the use docker ps -s.
Another way that I can think of is to expose kubelet/cadvisor metrics and use it with metrics scraper like Prometheus.
The kubelet exposes all of it’s runtime metrics, and all of the cAdvisor metrics, on a /metrics endpoint in the Prometheus exposition format. Note that kubelet also exposes metrics in /metrics/cadvisor, /metrics/resource and /metrics/probes endpoints. Disk metrics are generally available on the /stats/summary endpoint.
Additional info:
found an answer to my requirement - list pods disk usage running in a given node
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.pods[0] | "PodName: ", .podRef.name, "usedBytes: ", .containers[].rootfs.usedBytes'get node filesystem usage
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.usedBytes'get node imageFs usage
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.usedBytes'get node iNodes stats
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.fs.inodesFree'
kubectl get --raw /api/v1/nodes/<NODE_NAME>/proxy/stats/summary | jq '.node.runtime.imageFs.inodesFree'