I've been reading through the github docs for kube-prometheus-stack trying to figure out the correct way to configure ServiceMonitors and AlertManager.

kube-prometheus-stack points back to the prometheus-operator docs for Getting Started and Alerting.

Here's where I could use some pointers:

Regarding ServiceMonitors:

• Generally speaking, should there always be 1 ServiceMonitor object to 1 k8s Service?

In my values-override.yaml override that I'm passing in with the helm install, I specified the below configuration attempting to have Prometheus pick up any ServiceMonitor object created:

prometheus:
  prometheusSpec:
    serviceMonitorSelectorNilUsesHelmValues: false

• Once the ServiceMonitor is created does Prometheus pick up the change automatically?

Regarding Alerting:

• Should my global configuration items (like smtp_from, smtp_smarthost, etc) be added in a local values-override.yaml file that I pass in during helm install? (helm install -f values-overrides.yaml prometheus prometheus-community/kube-prometheus-stack) -- or should this configuration go in as part of the AlertmanagerConfig object?

• Speaking of AlertmanagerConfig - is the intention that there will only be ONE of these objects, or is it appropriate to create one for each alert route?

And finally at this point, my understanding is that the "routes" in the configuration behind AlertManager serve as the funnel through which any notifications will flow. So if any rule I create is triggered, AlertManager will find an appropriate matching route through which to send the the alert. This, in turn, means that there is no coupling between rules and alerts because the AlertmanagerConfig can be changed at any time to alter how alerts are sent out.

The learning curve for this seems rather steep... and I haven't even scratched the surface of scraping yet (pun intended). Any insight, confirmation, or general pointing-in-the-right-direction is GREATLY appreciated.