I am running Service monitors to gather metrics from pods. Then with the help of the Prometheus operator, I am using serviceMonitorSelector to catch those metrics in Prometheus. I see those metrics in Prometheus being collected.
Now, I am trying to export those custom metrics from Prometheus to AWS Cloudwatch. Does anyone have any idea how to do that? The end result is to set and alerting system with the help of Zenoss on Cloudwatch.
You have set up something like the prometheus-to-cloudwatch. You can run it in Kubernetes or on any server then make it scrape the same target that Prometheus is scraping. (prometheus-to-cloudwatch scrapes metrics from exporters or as a Prometheus client and not from the Prometheus server)
Then whatever you scrape will show up as metrics in Cloudwatch and then you can set alerts on those. For Zenoss you can use the AWS ZenPack and read the metrics from CloudWatch.
The Kubernetes Prometheus Operator automatically scrapes the services in your Kubernetes cluster and dynamically scrapes them as they get created, you will probably have check what targets are being scraped by Prometheus dynamically to configure what to scrape with prometheus-to-cloudwatch (Or you could build another operator; a prometheus-to-cloudwatch operator, but that will take time/work)
(There isn't such as thing as a scraper of the Prometheus server to CloudWatch either)