We have a Spring Boot application, deployed into K8s cluster (with 2 instances) configured with Micrometer exporter for Prometheus and visualization in Grafana.
I've implemented couple of additional Micrometer metrics, that report some information regarding business data in the database (PostgreSQL) and I could see those metrics in Grafana, however separately for each pod.
For our 2 pods in Grafana - I can see separate set of same metrics and the most recent value can be found by choosing (by label) one of the pods.
However there is no way to tell which pod reported the most recent values.
Is there a way to somehow always show the metrics values from the pod that was scraped last (ie it will contain the most fresh metric data)?
Right now in order to see the most fresh metric data - I have to switch pods and guess which one has the latest values.
(The metrics in question relate to database, therefore yielding the same values no matter the pod from which they are requested.)
In Prometheus, you can obtain the labels of the latest scrape using topk() and timestamp() function:
topk(1,timestamp(up{job="micrometer"}))This can then be used in Grafana to populate a (hidden) variable containing the instance name:
Name: instance
Type: Query
Query: topk(1,timestamp(up{job="micrometer"}))
Regex: /.*instance="([^"]*)".*/I advise to active the refresh on time range change to get the last scrape in your time range.
Then you can use the variable in all your dashboard's queries:
micrometer_metric{instance="${instance}"}EDIT: requester wants to update it on each data refresh
If you want to update it on each data refresh, it needs to be used in every query of your dashboard using AND logical operator:
micrometer_other_metric AND ON(instance) topk(1,timestamp(up{job="micrometer"}))vector1 AND vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped.