Divide groups with PromQL

9/10/2018

I want to calculate the actual Container CPU usage by dividing the number of used cores with the CPU limit (number of assignable cores). Accordingly I get two different metrics for 4 pods:

  • Number of used cores for each pod
  • Number of available cores for each pod

My problem:

I'd like to get the CPU usage for each container (number of used cores / number of available cores).

What I tried:

Each of these two queries return exactly what I want:

  1. Number of currently used cores for each pod:

(I am using label_replace because one metric uses pod_name as metric name and the other uses pod)

label_replace(sum(rate(container_cpu_usage_seconds_total{pod_name=~"rewe-bd-palantir-vernemq.*", container_name="vernemq"}[1m])) by (pod_name), "pod", "$1", "pod_name", "(.*)")

Response: https://monosnap.com/direct/6EPuLF59HBJaYsAmKG6CM0fRPyUXDk

  1. Number of available cores for each pod:

sum(kube_pod_container_resource_limits_cpu_cores{pod=~"rewe-bd-palantir-vernemq.*", container="vernemq", job="kubernetes-pods"}) by (pod)

Response: https://monosnap.com/direct/dRBfitwcxHIrTRYDmYHwV5YkomYJjH

This query didn't work (returned no data points):

label_replace(sum(rate(container_cpu_usage_seconds_total{pod_name=~"rewe-bd-palantir-vernemq.*", container_name="vernemq"}[1m])) by (pod_name), "pod", "$1", "pod_name", "(.*)") / sum(kube_pod_container_resource_limits_cpu_cores{pod=~"rewe-bd-palantir-vernemq.*", container="vernemq", job="kubernetes-pods"}) by (pod)

My question:

How could I achieve a query which returns the CPU usage (number of used cores / number of available cores) for each pod?

-- kentor
kubernetes
prometheus
promql

1 Answer

2/18/2019

You need to use the on() function as well. So it would be like this.

label_replace(sum(rate(container_cpu_usage_seconds_total{pod_name=~"rewe-bd-palantir-vernemq.*", container_name="vernemq"}[1m])) by (pod_name), "pod", "$1", "pod_name", "(.*)") / on(pod) sum(kube_pod_container_resource_limits_cpu_cores{pod=~"rewe-bd-palantir-vernemq.*", container="vernemq", job="kubernetes-pods"}) by (pod)
-- David Ferdinand
Source: StackOverflow