How to discover pods for prometheus to scrape

10/11/2018

I am trying to follow https://burhan.io/flask-application-monitoring-with-prometheus/ and make my pods discovered by Prometheus but I am not having any luck. Could someone see what I am doing wrong or debug it?

First to make sure my app is configured right...I configured it directly and saw the metrics in Prometheus.

- job_name: 'myapp'
        scheme: http
        static_configs:
        - targets: ['172.17.0.7:9090']

Next, I tried to do the discovery. This is how the deployment looks

kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp 
spec:
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 9090
...

and this is the prometheus config

  - job_name: 'kubernetes-pods'
        scheme: http
        metrics_path: /metrics
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_label_app]
          regex: myapp
          action: keep

but I don't see any metrics in Prometheus or any mention of myapp in Prometheus debug log. What am I missing?

-- suppandi g
kubernetes
prometheus

2 Answers

10/12/2018

My mistake. role had to be pod instead of node in the config.

-- suppandi g
Source: StackOverflow

10/12/2018

I see that you didn't define - api_server: 'https://kubernetes'. Make sure you define api-server in kubernetes_sd_config. Prometheus auto discovers services via api-server.

  • Please refer my previous question
  • Sample configuration in my repo here
  • Prometheus kubernetes_sd_config docs here
-- Veerendra Kakumanu
Source: StackOverflow