TL;DR Can I register multiple APIs to provide custom metrics in k8s?
We have a multi-tenanted cluster - with different deployments in different namespaces, and they each have their own monitoring solution (using Prometheus)
We want each namespace to be able to use the HPA to scale off of custom metrics from the Prometheus instance in their own namespace.
We have custom metrics configured using the Prometheus adapter (https://github.com/DirectXMan12/k8s-prometheus-adapter) in our dev environment and that works with the following deployed:
apiservice.apiregistration.k8s.io that is named v1beta1.custom.metrics.k8s.io and points to the service in the namespaceThe APIService doesn't get put in a namespace and is global - and the naming has to be v1beta1.custom.metrics.k8s.io which means we can only use a single prometheus for custom metrics - which breaks our usecase.
Any idea how to register more than one service as an API for custom_metrics? or another way to do this?
So I'm not sure this is the best answer - waiting for the community to respond before I say that - but it is definitely an answer.
You can't seem to register multiple services for a single API - so the answer is to aggregate the data into a single API.
Options for this include:
Federated Prometheus = scraping the /federated endpoint of each Prometheus and using that to server the custom scaling metrics. (https://prometheus.io/docs/prometheus/latest/federation/)
Thanos - (https://github.com/improbable-eng/thanos) Bit early days for production use maybe - so the squeamish may want to go for options one -but Thanos is giving me a unified querying layer over the top of every Prometheus, that I can attach the Adapter to. Pretty neat - nice project - worth checking out;