How to configure central Prometheus/grafana to monitor/scrape several k8s clusters

11/3/2021

is it possible to setup a prometheus/grafana running on centos to monitor several K8S clusters in the lab? the architecture can be similar to the bellow one, although not strictly required. Right now the kubernetes clusters we have, do not have prometheus and grafana installed. The documentation is not very much clear if an additional component/agent remote-push is required or not and how the central prometheus and the K8S need to be configured to achieve the results? Thanks.

Required architecture

-- Noune Zozanian
centralized
kubernetes
monitoring
prometheus

1 Answer

11/3/2021

You have different solution in order to implement your use case :

  1. You can use prometheus federation. This will allow you to have a central prometheus server that will scrape samples from other prometheus servers.
  2. You can use remote_write configuration. This will allow you to send your samples to a remote endpoint (and then eventually scrape that central endpoint). You'll also be able to apply relabeling rules with this configuration.
  3. As @JulioHM said in the comment, you can use another tool like thanos or Cortex. Those tools are great and allow you to do more stuff than just writing to a remote endpoint. You'll be able to implement horizontal scalling of your prometheus servers, long-term storage, etc.
-- Marc ABOUCHACRA
Source: StackOverflow