So I have 3 clusters of Kubernetes. In one of them, I have deployed Prometheus (and some other apps which I am monitoring). But I also want to monitor Applications (spring boot) that are in other clusters, at the pod level.
I have looked into different solutions like using the hierarchial-federation deployment for Prometheus but that really looks a bit messy. Many have suggested Service Discovery, but there are no resources for the same to monitor outside the cluster.
I am still a beginner in K8, so can you guys suggest to me how to do Pod level monitoring in Prometheus for applications that are deployed in a different Kubernetes cluster?
You certainly cannot use service monitors and pod monitors as long as Prometheus does not reside in same cluster as your apps. If your applications already expose prometheus style metrics, you may want to take a look at
Eg:
additionalScrapeConfigs:
- job_name: <a job name>
honor_labels: true
honor_timestamps: true
scrape_interval: 10s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets: ["app1:9102", "app2:9102", "app3:9102"]This solution is limited since it needs configuration at Prometheus side, so will not auto discover your metrics endpoints and scale.