K
Q

Combine Grafana metrics with mismatched labels

11/26/2018

I have two metrics (relating to memory usage in my Kubernetes pods) defined as follows:

  1. kube_pod_container_resource_limits_memory_bytes{app="kube-state-metrics",container="foo",instance="10.244.0.7:8080",job="kubernetes-endpoints",kubernetes_name="kube-state-metrics",kubernetes_namespace="monitoring",namespace="test",node="aks-nodepool1-25518080-0",pod="foo-cb9bc5fb5-2bghz"}
  2. container_memory_working_set_bytes{agentpool="nodepool1",beta_kubernetes_io_arch="amd64",beta_kubernetes_io_instance_type="Standard_A2",beta_kubernetes_io_os="linux",container_name="foo",failure_domain_beta_kubernetes_io_region="westeurope",failure_domain_beta_kubernetes_io_zone="1",id="/kubepods/burstable/pod5b0099a9-eeff-11e8-884b-ca2011a99774/eeb183b21e2b3226a32de41dd85d7a2e9fc8715cf31ea7109bfbb2cae7c00c44",image="@sha256:6d6003ba86a0b7f74f512b08768093b4c098e825bd7850db66d11f66bc384870",instance="aks-nodepool1-25518080-0",job="kubernetes-cadvisor",kubernetes_azure_com_cluster="MC_test.planned.bthbygg.se_bthbygg-test_westeurope",kubernetes_io_hostname="aks-nodepool1-25518080-0",kubernetes_io_role="agent",name="k8s_foo_foo-cb9bc5fb5-2bghz_test_5b0099a9-eeff-11e8-884b-ca2011a99774_0",namespace="test",pod_name="foo-cb9bc5fb5-2bghz",storageprofile="managed",storagetier="Standard_LRS"}

I want to combine these two into a percentage, by doing something like

container_memory_working_set_bytes{namespace="test"}
  / kube_pod_container_resource_limits_memory_bytes{namespace="test"}

but that gives me no data back, presumably because there are no matching labels to join the data sets on. As you can see, I do have matching label values, but the label names don't match.

Is there somehow I can formulate my query to join these on e.g. pod == pod_name, without having to change the metrics at the other end (where they are exported)?

-- Tomas Aschan
grafana
kubernetes
prometheus

Similar Questions

How to establish communication between two private GCP clusters which are located in different regions?
Scaling down old replicasets while deploying requires data migration
PersistentVolumeClaim fails to create on Alicloud Kubernetes
pod does not become running within 10 min, how to exit it directly (will never become running)
Display completed Kubernetes ConfigMap

1 Answer

11/26/2018

You can use PromQL label_replace function to create a new matching label from the original labels.

For instance, you can use the below expression to add a container_name="foo" label to the first metric which can be used to do the join:

label_replace(
 kube_pod_container_resource_limits_memory_bytes,
 "container_name", "$1", "container", "(.*)")

You can use the above patern to create new labels that can be used for the matching.

-- yamenk
Source: StackOverflow