Have a Prometheus operator on GKE and some ConfigMap with Prometheus rules, created by me. Today I figured out, that I can't change/delete that ConfigMap anymore. Each time it gets recreated in the previous state. Back in days, it wasn't immutable.
What can be the cause of that?
Prometheus Operator acts on CRDs. These objects are continually watched, and any drift configuration will trigger a config-reload.
The operator is intended to fully control the ConfigMap; if you directly edit it, the config-reloader will eventually revert your changes to match the CRD configs.
The correct way to edit your rules is by changing the PrometheusRule object. Your changes will be caught by the operator, which will update the ConfigMap and trigger a config-reload.