Grafana variable query is not retrieving all endpoints' from a Prometheus pod target (but gets some of them)

5/25/2021

While maintaining a simple set of a Grafana and a Prometheus pod, plus a few others, within a cluster on Azure Kubernetes Services (AKS)—I ran into an issue where a "instance" variable query set up in Grafana only retrieves any data from several, out of 52, VM's/instances in a Prometheus scrape job.

There are a couple other Grafana variables that can arbitrarily nest / be nested inside of the "instance" variable, but switching the order of them did nothing. the variables, including "instance" and "environment" which do not depend on each other and I tried switching

To elaborate, this is bizarre because I am getting perfect data for 11 out of the 52 VM's/instances while the others are not being populated in the "instance" Grafana variable as they should be. Maybe this is a backend issue, but have not found any oddities while probing around with kubectl.

Thank you!

-- Jackalakalaka
azure-aks
grafana
grafana-templating
kubernetes
prometheus

1 Answer

6/7/2021

All the 52 instances were not sending the windows_iis_requests_total metric.Hence the label_values gave the incomplete response.

-- rohatgisanat
Source: StackOverflow