Prometheus service endpoint scrapes all the ports of a service instead of just annotated ports

12/11/2018
Kubernetes version - v1.11.2
Prometheus helm chart version - 6.7.0     

I have my service running on 2 ports - 80 and 9000 Now I only need to monitor port 80 and I used below configuration to achieve that.

- job_name: '<service-name>'
          honor_labels: true
          kubernetes_sd_configs:
            - role: service
          relabel_configs:
            - source_labels: [__meta_kubernetes_service_label_app]
              action: keep
              regex: <service-name>
            - source_labels: [__meta_kubernetes_service_name]
              action: replace
              target_label: kubernetes_name

The above solution adds both the service endpoint in Prometheus

http://<service-name>.default.svc:80/metrics
http://<service-name>.default.svc:9000/metrics

To scrape only port 80 I added below config but its not able to scrape any service endpoints now.

- source_labels: [__meta_kubernetes_service_port_number]
              action: keep
              regex: 8\d{1}

Is there a way to only restrict specific port numbers?

-- Darshan Deshmukh
kubernetes
prometheus
prometheus-alertmanager
prometheus-operator

1 Answer

7/19/2019

I had a similar issue, specifying the port in the relabel_configs worked for me.

relabel_configs: 
  - source_labels : [__meta_kubernetes_pod_label_app,__meta_kubernetes_pod_container_port_number]
    action: keep
    regex: myapp;8081

After this my service was getting scraped only for port 8081

-- akash gautam
Source: StackOverflow