I am using prometheus operator to monitor my Kubernetes cluster. I want to change the scrape_interval for some targets dynamically (increase and decrease it when needed at runtime).
Any suggestions to do that?
Thanks
you can change that on the fly-
find servicemonitor ( on prometheus UI it is target ) as-kubectl get servicemonitor -n <NAMESPACE>.
k get servicemonitor -n <NAMESPACE> <SERVICEMONITOR_NAME> -o yaml | sed 's/interval: [0-9][0-9]s/interval: 50s/' | kubectl replace -f -
To reload automatically, prometheus operator has config-reloader, you can install that as a container alongside your prometheus.