I am setting up Prometheus on GKE. I have some VMs external to the GKE cluster running node exporter that I wish to also retrieve machine metrics.

I setup gce_sd_configs in Prometheus and it finds out about the existance external VMs. However, it cannot retrieve the metrics from those VMs as access from the pod running prometheus to those VMs appears to be blocked.

I have attempted to setup some firewall rules. All my research so far talks about creating a "headless service" to access services outside of the Kubernetes cluster. However, this doesn't feel scalable to do if I want to monitor metrics for all my VMs running outside of Kubernetes.

Has anyone experience with running this setup?