Kubernetes: Comparing RSS Memory Usage of Pods and Pod Memory Requirements in Prometheus / PromQL

3/5/2019

We have a metric container_memory_rss from cadvisor and a metric kube_pod_container_resource_requests_memory_bytes from Kubernetes itself.

Is it possible to join the metrics against one another so that we can directly compare the ratio of both metrics? More specifically I'd like to basically 'join' the following metrics:

sum(kube_pod_container_resource_requests_memory_bytes) by (pod, namespace)
sum(container_memory_rss) by (container_label_io_kubernetes_pod_name, container_label_io_kubernetes_pod_namespace)

The 'join' would be on pod name and namespace.

Can PromQL do this, given that the label names vary?

-- Johannes Gehrs
cadvisor
kubernetes
prometheus
promql

1 Answer

3/5/2019

You can use the label_replace function to modify labels on one side of the expression such that they match:

  sum by (pod_name, namespace) (container_memory_rss) 
/ 
  sum by (pod_name, namespace) (
    label_replace(
      kube_pod_container_resource_requests_memory_bytes, 
      "pod_name", "$1", "pod", "(.*)"
    )
  ) 
-- tom.wilkie
Source: StackOverflow