Monitor Kubernetes Cluster from other Kubernetes cluster with Prometheus

12/2/2019

I have run several Kubernetes Clusters on Azure AKS, so I intend to create Centralized Monitoring Cluster run on other Kubernetes cluster with Prometheus-Grafana.

Centralized Monitoring

My idea is that:

  • Isolating & centralizing monitoring cluster.
  • In case of any cluster is downed, Monitoring cluster is still alive to inspect downed cluster.
  • Run cross-cloud provider Kubernetes (if available)

I'm confusing about connecting clusters, network, ingress, how does Prometheus discovery, pull metric from outside cluster...

Is there any best practice, instruction for my usecase. Thank you!

-- Khoa
cloud
kubernetes
monitoring
prometheus

1 Answer

12/2/2019

Yes, Prometheus is a very flexible monitoring solution wherein each Prometheus server is able to act as a target for another Prometheus server. Using prometheus federation, Prometheus servers can scrape selected time series data from other Prometheus servers.

A typical Prometheus federation example configuration looks like this:

- job_name: 'federate'
  scrape_interval: 15s

  honor_labels: true
  metrics_path: '/federate'

  params:
    'match[]':
      - '{job="prometheus"}'
      - '{__name__=~"job:.*"}'

  static_configs:
    - targets:
      - 'source-prometheus-1:9090'
      - 'source-prometheus-2:9090'
      - 'source-prometheus-3:9090'
-- P Ekambaram
Source: StackOverflow