I have a basic prometheus.yml file in my environment i.e ..
###
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: prometheus-core
data:
prometheus.yml: |
global:
scrape_interval: 10s
scrape_timeout: 10s
evaluation_interval: 10s
rule_files:
- '/etc/prometheus-rules/*.rules'
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090'] Now if I add new nodes to my environment, my prometheus.yml file should automatically get updated and add the nodes to the targets below i.e.
###
apiVersion: v1
kind: ConfigMap
metadata:
creationTimestamp: null
name: prometheus-core
data:
prometheus.yml: |
global:
scrape_interval: 10s
scrape_timeout: 10s
evaluation_interval: 10s
rule_files:
- '/etc/prometheus-rules/*.rules'
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# Override the global default and scrape targets from this job every 5 seconds.
scrape_interval: 5s
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090','12.10.17.6:9100','12.10.17.19:9100'] Can any one suggest how I can achieve this ?
Prometheus supports a Kubernetes service discovery mechanisms, see the documentation for details.
So instead of the static_configs section, you should add a section similar to this:
scrape_configs:
- job_name: 'kubernetes-service-endpoints'
kubernetes_sd_configs:
- role: endpoints
...
See this example configuration file for how it's done.