I have a Fluentd daemonset on the k8s cluster with 3 nodes. I would like to get a value which represents percentage which gives me an information on how much CPU (in %) on each node, fluentd pod occupies at this moment.
What would be the way to do it in Prometheus?
Thanks.
You would want to use the container_cpu_usage_seconds_total
query in Prometheus.
Like so:
sum (rate (container_cpu_usage_seconds_total{}[5m])) by (container_name)
This will return the CPU usage of all the pods by
container name in the system.
You can apply some filters as well to fine grain the output. For example:
sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*"}[5m])) by (container_name)
The above query will return the CPU usage with pods that matches containers name starting with fluentd
You can divide the usage of those pods by the total cpu cores of your cluster to find out the usage in percentage, like so:
sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*"}[5m])) / sum (machine_cpu_cores{}) * 100
Finally, in order to get percentage of total cpu cores usage on specific node for specific container_name, you would add additional filter: instance="INSTANCE_NAME":
sum (rate (container_cpu_usage_seconds_total{container_name=~"fluentd.*", instance="INSTANCE_NAME"}[5m])) / sum (machine_cpu_cores{}) * 100
N.B: Depending on K8S version, the fields returned by the container_cpu_usage_seconds_total
query may vary. On some system, the name of the containers is represented by the container_name
field, wheres on some system it is container
.