I am trying to figure out how to get pod labels into the metric tags from kubelet metrics using prometheus-stack. In our environment, we need to hash pod names (due to length limitations) so our app name, env, and unit name are saved in pod labels.
We are using prometheus-stack (helm installation) to collect metrics from kubelet (/metrics, /metrics/cadvisor) and due to the lack of pod labels in metrics tags, it's difficult to know which metric belongs to which application.
Prometheus-stack is using sd_kubernetes_config with endpoint rule to collect kubelet metrics, where __meta tags for pod labels cannot be used. Is there another way how to get that labels in metric tags?
I also tried to collect pod_labels metric using kubeStateMetrics, where I can get metric that contains pod labels, but I cannot figure out how to display both metrics in a way that metric from cadvisor will show its value and metric from kubeStateMetrics will be used to display its labels (in Prometheus graph).
Thanks for any advice.
As far as I know you can really use filtering metrics based on pod labels. Look at the original answer:
You can use
+operator to join metrics. Here,group_left()will include the extra label:label_sourcefrom the right metrickube_pod_labels. The metric you're joining is forced to zero ( i.e.0 * kube_pod_labels) so that it doesn't affect the result of first metric.
(
kube_pod_info{namespace="test"}
)
+ on(namespace) group_left(label_source)
(
0 * kube_pod_labels
)
In fact, metrics then can be long and nasty. However, it is an effective method in prometheus to create what you expect, which is kubelet metrics with pod labels.
Look also at this guide - it is described how to join Prometheus metrics by label with PromQL and this another guide - How to use PromQL joins for more effective queries of Prometheus metrics at scale.