While maintaining a simple set of a Grafana and a Prometheus pod, plus a few others, within a cluster on Azure Kubernetes Services (AKS)—I ran into an issue where a "instance" variable query set up in Grafana only retrieves any data from several, out of 52, VM's/instances in a Prometheus scrape job.
There are a couple other Grafana variables that can arbitrarily nest / be nested inside of the "instance" variable, but switching the order of them did nothing.
To elaborate, this is bizarre because I am getting perfect data for 11 out of the 52 VM's/instances while the others are not being populated in the "instance" Grafana variable as they should be. Maybe this is a backend issue, but have not found any oddities while probing around with kubectl
.
Thank you!
All the 52 instances were not sending the windows_iis_requests_total metric.Hence the label_values gave the incomplete response.