Get value of other label while applying filter on one label in PromQL

2/2/2021

I have a kubernetes cluster hosting multiple tenants. I want to filter out the nodes belonging to a particular tenant. For example:

Querying kube_node_labels to prometheus results in the following output:

kube_node_labels{ node="ip-10-139-110-166.ec2.internal", label_tenant: "tenant1" }
kube_node_labels{ node="ip-10-139-110-167.ec2.internal", label_tenant: "tenant2" }
kube_node_labels{ node="ip-10-139-110-168.ec2.internal", label_tenant: "tenant1" }

Is there a way I can just get all the nodes belonging to a particular tenant as array? For e.g. ip-10-139-110-166.ec2.internal, ip-10-139-110-168.ec2.internal for tenant1

Basically, what's the equivalent of select node from kube_node_labels where label_tenant="tenant1" in PromQL?

-- Aman
grafana
kubernetes
monitoring
prometheus
promql

1 Answer

2/3/2021

I was able to achieve it with the following query: label_values(kube_node_labels{label_tenant="tenant1"}, node)

-- Aman
Source: StackOverflow