In the system we are building, we need to monitor components(Kafka Connect,Spark..etc) runs in k8s cluster using Prometheus. Since k8s cluster runs in a different network, Prometheus server cannot scrape these components directly. For this, we are going to run a PushGateWay in our k8s cluster.

I come up with the following design :

1. Custom built "Push Proxy" component reads "kube-api-service" to get the list of pods for each component we want to monitor.(Also periodically checks pod health/check new pods..etc)
2. The "Push Proxy" periodically scraps Exporters in each pod. (Calling /metrics endpoint of each pod). Store these scraped matrices in a buffer.
3. "Push Proxy" periodically push the buffered matrices to the PushGateWay.
4. This PushGateWay is accessible from Prometheus. Prometheus scrapes matrices periodically from the PushGW.

Questions are:

1. is there any component like the "Push Proxy" we are planning on building?
2. Is this a valid design to handle this problem.