We have a prometheus operator setup running in our kubernetes cluster. We have many different namespace for teams/users. As of now we have a static alertmanager config that sets up routes based on the namespace labels and sends emails to the respective teams.
I wonder if there is an elegant way to dynamically configure such routes, for example, using annotations or labels in every namespace defining the email addresses.
My two ideas to achieve this are:
Use alertmanager webhook and send the alerts to a service that frequently scrapes the labels or watches namespace objects and then sends the emails to the teams.
Use a custom admission controller/webhook that listens on namespace changes and modifies the alertmanager configmap.
I prefer the second solution so far as it makes use of the alertmanager email templates and in general, sounds like the cleaner solution to me.
Perhaps there is a better way to solve this or already existing solutions that I missed?