K
Q

How to update Prometheus config in k8s cluster

11/9/2018

I have running Prometheus in k8s. Could you please advice how can I change running config prometheus.yaml in cluster? I just want simply to change:

scrape_configs:
- job_name: my-exporter
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http

How can I do this?

Thanks.

-- JBB
config
kubernetes
prometheus

Similar Questions

Bind nginx-ingress to static IP Address
Kubernetes GKE in GCP replicas number doesn't match
helm install stable/gocd returns an error
How to get kubernetes / minikube env variable in vue js
(Kubernetes + Docker) Skaffold keeps terminating my deployment files : Error: could not stabilize within 2m0s: context deadline exceeded
How to expose kubernetes web server to port 80
Kubernetes Node 14 pod restarts and terminated with exit code 0
EKS Ingress ALB add HTTP listener for some services and HTTPS for others

1 Answer

11/17/2018

The recommended way is to provide the prometheus.yml via a ConfigMap. That way changes in the ConfigMap will be propagated into the pod that consumes the configMap. However, that is not enough for prometheus to pick up the new config.

Prometheus supports runtime reload of the config, so that you don't need to stop prometheus in order to pickup the new config. You can either do that manually by sending a POST request as described in the link above, or automate this process by having a sidecar container inside the same prometheus pod that watch for updates to the config file and does the reload POST request.

The following is an example on the second approach: prometheus-configmaps-continuous-deployment

-- yamenk
Source: StackOverflow