My scenarios is like below
There is one prometheus server and i want to monitor my devops servers, dev deployment (which is a kubernetes cluster with 10 pods), qa deployment and prod deployment.
I want to monitor all those deployments and devops servers by central prometheus server.
I have seen couple of articles in this context. but they all refer setup prometheus on kubernetes cluster.
Looking help on how to configure k8s clusters on separate prometheus server.
Based on question on Prometheus Github and it's documentation, the only optimal way to get data from several K8s cluster is to use Federation.
You will have several Prometheus servers anyway (not only because of storing data, but also because, as example, processing custom resource which used for define check, see design document about how it works), but in Federation mode it is possible to use it's data together.
So, you can configure, as example, Hierarchical Federation of Prometheus. In that mode you can use your separate Prometheus as a higher-level server, which will collect data from servers on the cluster.
You can configure you server for get metrics from /federate endpoint of your Prometheus servers working on K8s clusters.
From article:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 'source-prometheus-1:9090'
- 'source-prometheus-2:9090'
- 'source-prometheus-3:9090'