K
Q

Prometheus Server with Dynamic Rules

1/31/2021

I am currently working on a project that uses the Prometheus Server (Not the Prometheus Operator). We're looking to introduce a way of modifying the PrometheusRules without having to redeploy it.

I'm completely new to containers and Kubernetes and a little over my head so I'm hoping someone could let me know if I'm wasting my time trying to make this work.

What I have thought of doing so far is:

  1. Store the PrometheusRules in a configmap.
  2. Apply the configmap of rules to the Prometheus Server configuration.
  3. Create a sidecar to the Prometheus Server that can modify this configMap.
  4. The sidecar will have an API exposed so users will have CRUD functionality for the rules.
  5. When successful modifying a rule, the sidecar will trigger the reload endpoint on the Prometheus Server that causes it to reload its configuration file without having to restart the container.

Thanks

-- Donncha
configmap
kubernetes
monitoring
prometheus

Similar Questions

Upgrade with Helm meet failure seems as configmap limitation exceed
how to refresh all kubernetes pods with actuator in spring boot config server
Kubernetes 1.13, CoreDNS - cluster curl service?
How to access nginx ingress controller
Can flink run mutliple same jobs to achieve pseudo dynamic scaling?

1 Answer

1/31/2021

Your initial use case seems valid, though I would say there are better ways of achieving this.

For points 1,2 I would suggest you use the Prometheus Helm Chart for ease of use, better config management and deployment. This would keep track of Prometheus configuration as one single unit rather than maintaining the rule files separately.

For point 3,4:- Making direct untracked changes to the live configuration does not seem safe or secure. Using the helm chart mentioned above I would suggest you make the changes before deploying to the cluster (Use VCS like Git to track changes)

Best case scenario:- Also setup CI/CD pipelines to deploy changes instantly.

Use the reload API as mentioned to reload the new released config.

Explore more about Helm

-- rohatgisanat
Source: StackOverflow