I tried file based service discovery ,But everytime when I change the configmap(which contains static target), I am deleting prometheus pod manually to get config changes. Is there any way that prometheus can get config changes automatically without deleting the prometheus pod? any help on this issue?
I am installing prometheus-operator using helm chart target.json file
[
{
"labels": {
"app": "web",
"env": "dev"
},
"targets": [
"web.dev.svc.cluster.local"
]
}
]```
command I used to create configmap
kubectl create cm static-config --from-file=target.json -n monitoring
prometheus-operator.yaml
```volumes:
- name: config-volume
configMap:
name: static-config
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus/config
additionalScrapeConfigs:
- job_name: 'file-based-targets'
file_sd_configs:
- files:
- '/etc/prometheus/config/target.json'```Prometheus reloads file_sd_configs automatically using file watches according to the documentation:
It reads a set of files containing a list of zero or more <static_config>s. Changes to all defined files are detected via disk watches and applied immediately. Files may be provided in YAML or JSON format. Only changes resulting in well-formed target groups are applied.
https://prometheus.io/docs/prometheus/latest/configuration/configuration/#file_sd_config
If you need to add more target files you can use a wildcard for the files, for example:
scrape_configs:
- job_name: 'file-based-targets'
file_sd_configs:
- files:
- '/etc/prometheus/targets/*.json'If you still need reloading from configmaps you can add another container to the Prometheus CRD and use either the built-in prometheus-operator/prometheus-config-reloader (this is how the Prometheus configuration and rules are reloaded) or one of the following: