We have configured Prometheus via helm chart and we would like to see the configuration and rules for Prometheus updated on the fly (hot reload).

Our design approach is something like this, 1. Have our application generate the alerts in a yaml file. 2. Load those alerts from a yaml file into Prometheus as rules. 3. Upon modifications to our application generated YAML file, update the prometheus rules as well. 4. Be able to tweak the prometheus configuration on the fly too.

After researching a bit , found following options: 1. If I use the values.yaml way of handling this, then that is more of static/default configuration to be used upon installation. And doesn't allow to add/minus alerts and change basic configuration at runtime. 2. If I use the PrometheusRules CRD way, not sure how to generate the PrometheusRules object using the custom alert yaml file from our application. 3. Upon changes to our yaml file, how to update the PrometheusRules object at runtime.

Not sure, if instead of depending upon prometheus operator , we should depend upon prometheus itself for configuration reload etc.

Any suggestions would be greatly appreciated?