I have two kubernetes clusters who do not talk to one another in any way. The idea is to maintain one prometheus instance(in another 3rd cluster) that can scrape endpoints from both the clusters.
I created a service account in each cluster, gave it cluster role & clusterrolebinding and took an yaml file of the secret. I then imported the same secret in the 3rd cluster where I have prometheus running. Using these mounted secrets, I was able to pull data from all pods in cluster 1 and 2.
Are there any better options to achieve this usecase? I am in a way transferring secrets from one cluster to another to get the same ca.crt and token.
I think it is not safe to share secrets between clusters.
What about federation prometheus, one prometheus instance can export some data, which can be consumed by external prometheus instance.
For example, a cluster scheduler running multiple services might expose resource usage information (like memory and CPU usage) about service instances running on the cluster. On the other hand, a service running on that cluster will only expose application-specific service metrics. Often, these two sets of metrics are scraped by separate Prometheus servers.
Or deploy some exporter, which can be consumed by external prometheus. e.g. https://github.com/kubernetes/kube-state-metrics (but it is not providing cpu/memory usage of pods)