I have deployed Prometheus on Kubernetes and I provide the prometheus.yml configuration file as a config map resource. The file is mounted in the Prometheus pod as volume.
After the config map is changed in the cluster, I hit the Prometheus server endpoint with an empty POST request, in order to reload it (as described in the documentation)
When I make changes to the config map however and redeploy it, I experience a 'lag' of around 30 secs until the prometheus.yml file is updated inside the pod.
I read here that this is expected.
However, there are projects that try to remedy this, eg
These, as I understand, kill the pods and replace them in order to update the configuration.
My question is, is there a way to make a 'hot' reconfiguration?
Essentially speed up on-demand the volume update inside the pod, without the need of killing any pod.
I'm using right now reloader and the pods are killed always but they are recreated immediately after you change a configmap associated.
But you cannot avoid the fact that you need to kill the pod to be able to re-mount the volume with the new config and also it depends on your workload if is able to load a new configuration without stopping the main process inside the container.