Monitor Kubernetes Cluster from other Kubernetes cluster with Prometheus
12/2/2019
I have run several Kubernetes Clusters on Azure AKS, so I intend to create Centralized Monitoring Cluster run on other Kubernetes cluster with Prometheus-Grafana.
My idea is that:
- Isolating & centralizing monitoring cluster.
- In case of any cluster is downed, Monitoring cluster is still alive to inspect downed cluster.
- Run cross-cloud provider Kubernetes (if available)
I'm confusing about connecting clusters, network, ingress, how does Prometheus discovery, pull metric from outside cluster...
Is there any best practice, instruction for my usecase. Thank you!
-- Khoa
cloud
kubernetes
monitoring
prometheus
Similar Questions
1 Answer
12/2/2019
Yes, Prometheus is a very flexible monitoring solution wherein each Prometheus server is able to act as a target for another Prometheus server. Using prometheus federation, Prometheus servers can scrape selected time series data from other Prometheus servers.
A typical Prometheus federation example configuration looks like this:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 'source-prometheus-1:9090'
- 'source-prometheus-2:9090'
- 'source-prometheus-3:9090'-- P Ekambaram
Source: StackOverflow