When we use Kubernetes for production and we have a scaled application with many pods and publish as a service, every single metrics fetching request of Prometheus is routed to a pod with a random of selection. In this situation, results are not true for monitoring.
In a moment we need all pods metrics (for example 10 pod) and it's not possible by calling a Kubernetes Service endpoint! Is there any solution for this problem?
You can configure your kubernetes_sd_configs so it scrapes the pods individually and not just the service. To do that, set the role to pod, like this:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: podSee this blog post for a full config example.