K
Q

How to get CPU and memory usage of pod in percentage using promethus

8/10/2020

I want to display pod details in the following format using promql/Prometheus.

Image1

Furthermore, I want to display CPU and memory utilization of application/component in below format using promql

Image2

promql query: sum(container_memory_working_set_bytes) by (pod)

I can get the consumed memory by pod using above query.

How to calculate percentage of memory used ? I am not able to fetch memory limit of stateful pod using promql Could you please suggest any query/API details ?

-- Tejas
grafana
kubernetes
monitoring
prometheus
promql

Similar Questions

Kubernetes: unreachable backend: read udp 10.200.0.9:46159->183.60.83.19:53: i/o timeout
My kubernetes pods restarting randomly with sandbox changed event reason, causing downtime of my services
Kubernetes LoadBalancer Service returning empty response
Difference between metalLB and NodePort
How to create a single MongoDB Container in Kubernates Cluster

3 Answers

9/9/2020

For CPU percentage 

max by (pod) (sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",container_name!="POD",container_name!="",container!="monitoring-daemon"}[5m])) / sum(kube_pod_container_resource_limits{namespace="$namespace", resource="cpu"})) * 100

For Memory percentage

avg((avg (container_memory_working_set_bytes{pod="<Podname>"}) by (container_name , pod ))/ on (container_name , pod)(avg (container_spec_memory_limit_bytes>0 ) by (container_name, pod))*100)
-- Tejas
Source: StackOverflow
11/4/2020

I use this for memory: 

round(max by (pod)(max_over_time(container_memory_usage_bytes{namespace="$namespace",pod=~".*" }[5m]))/ on (pod) (max by (pod) (kube_pod_container_resource_limits)) * 100,0.01)

And this for CPU : 

max by (pod) (sum(rate(container_cpu_usage_seconds_total{namespace="$namespace",container_name!="POD",container_name!="",container!="monitoring-daemon"}[5m])) / sum(kube_pod_container_resource_limits{namespace="$namespace", resource="cpu"})) * 100
-- Nicolas Roux
Source: StackOverflow
3/25/2022

Per-pod CPU usage in percentage (the query doesn't return CPU usage for pods without CPU limits)

100 * max(
  rate(container_cpu_usage_seconds_total[5m])
    / on (container, pod)
  kube_pod_container_resource_limits{resource="cpu"}
) by (pod)

The kube_pod_container_resource_limits metric can be scraped incorrectly if scrape config for kube-state-metrics pod is improperly configured. In this case the original pod label for this metric is moved to the exported_pod label because of honor_labels behavior - see these docs for details. In this case label_replace function must be used for moving exported_pod label to pod label:

100 * max(
  rate(container_cpu_usage_seconds_total[5m])
    / on (container, pod)
  label_replace(kube_pod_container_resource_limits{resource="cpu"}, "pod", "$1", "exported_pod", "(.+)")
) by (pod)

Per-pod memory usage in percentage (the query doesn't return memory usage for pods without memory limits)

100 * max(
  container_memory_working_set_bytes
    / on (container, pod)
  kube_pod_container_resource_limits{resource="memory"}
) by (pod)

If the kube_pod_container_resource_limits metric is scraped incorrectly as mentioned above, then the label_replace function must be used for moving exported_pod label value to pod:

100 * max(
  container_memory_working_set_bytes
    / on (container, pod)
  label_replace(kube_pod_container_resource_limits{resource="memory"}, "pod", "$1", "exported_pod", "(.+)")
) by (pod)
-- valyala
Source: StackOverflow