How to calculate percentage of specific pod CPU usage on each node?

11/16/2019

I have a Fluentd daemonset on the k8s cluster with 3 nodes. I would like to get a value which represents percentage which gives me an information on how much CPU (in %) on each node, fluentd pod occupies at this moment.

What would be the way to do it in Prometheus?

Thanks.

-- Bakir Jusufbegovic
kubernetes
prometheus
promql

1 Answer

11/17/2019

You would want to use the container_cpu_usage_seconds_total query in Prometheus.

Like so:

sum (rate (container_cpu_usage_seconds_total{}[5m])) by (container_name)

This will return the CPU usage of all the pods by container name in the system.


You can apply some filters as well to fine grain the output. For example:

sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*"}[5m])) by (container_name)

The above query will return the CPU usage with pods that matches containers name starting with fluentd


You can divide the usage of those pods by the total cpu cores of your cluster to find out the usage in percentage, like so:

sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*"}[5m])) / sum (machine_cpu_cores{}) * 100

Finally, in order to get percentage of total cpu cores usage on specific node for specific container_name, you would add additional filter: instance="INSTANCE_NAME":

sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*", instance="INSTANCE_NAME"}[5m])) / sum (machine_cpu_cores{}) * 100

N.B: Depending on K8S version, the fields returned by the container_cpu_usage_seconds_total query may vary. On some system, the name of the containers is represented by the container_name field, wheres on some system it is container.

-- aniskhan001
Source: StackOverflow