Is it possible to aggregate 2 gauge metrics (i.e. kube_pod_labels and kube_pod_container_resource_requests_cpu_cores) so that by executing the query both resulting elements would be combined (i.e. all pod labels as resulting element and request_cpu_cores as value)?
Query for cpu request value looks like this sum by (namespace, pod) (kube_pod_container_resource_requests_cpu_cores{cluster="my-cluster"})
Actual result:
{namespace="my-ns",pod="my-pod"} 0.05Query for pod labels is kube_pod_labels{label_foo="bar"}
Actual result:
kube_pod_labels{cluster="my-cluster",label_foo="bar",label_app="my-app-label",namespace="my-ns",pod="my-pod",service="my-svc"} 1I have tried using a left joint but it seems that grouping by a given label (pod, namespace etc) is required as explained in this https://www.robustperception.io/left-joins-in-promql.
With a multiplication operator * is possible to obtain the desired result set but the set would only contain labels specified in the by clause. Example query:
group by (namespace,pod) (kube_pod_labels{label_foo="bar",cluster="my-cluster"}) * sum by (namespace, pod) (kube_pod_container_resource_requests_cpu_cores{cluster="my-cluster"})Example result:
{namespace="my-ns",pod="my-pod"} 0.05What I am trying to obtain is a resulting set containing all labels without having to filter by an arbitrary label/value
The desired result by joining the 2 queries should be:
{cluster="my-cluster",label_foo="bar", label_app="my-app-label",namespace="my-ns",pod="my-pod",service="my-svc"} 0.05This can be achieved with a combination of the following:
label_replace query function: For each timeseries in v, label_replace(v instant-vector, dst_label string, replacement string, src_label string, regex string) matches the regular expression regex against the value of the label src_label. If it matches, the value of the label dst_label in the returned timeseries will be the expansion of replacement, together with the original labels in the input. Capturing groups in the regular expression can be referenced with $1, $2, etc. If the regular expression doesn't match then the timeseries is returned unchanged. https://prometheus.io/docs/prometheus/latest/querying/functions/#label_replace
multiplication * operator and group_left() modifier: Many-to-one and one-to-many matchings refer to the case where each vector element on the "one"-side can match with multiple elements on the "many"-side. This has to be explicitly requested using the group_left or group_right modifier, where left/right determines which vector has the higher cardinality. https://prometheus.io/docs/prometheus/latest/querying/operators/
Example query:
label_replace(kube_pod_labels{},"label","$1","label_", "(.+)")
* on (cluster,namespace, pod) group_left()
(sum by (cluster,namespace, pod) (kube_pod_container_resource_requests_cpu_cores{}))Note that: If the regular expression doesn't match then the timeseries is returned unchanged. In this case the regular expression does not match - hence the full set of labels is return unchanged.
Example result:
{cluster="my-cluster",label_foo="bar", label_app="my-app-label",namespace="my-ns",pod="my-pod",service="my-svc"} 0.05Felipe provided a valuable hint on how to achieve this result in a comment for the original question.