I am trying to follow https://burhan.io/flask-application-monitoring-with-prometheus/ and make my pods discovered by Prometheus but I am not having any luck. Could someone see what I am doing wrong or debug it?
First to make sure my app is configured right...I configured it directly and saw the metrics in Prometheus.
- job_name: 'myapp'
scheme: http
static_configs:
- targets: ['172.17.0.7:9090']
Next, I tried to do the discovery. This is how the deployment looks
kind: Deployment
metadata:
name: myapp
labels:
app: myapp
spec:
template:
metadata:
labels:
app: myapp
spec:
containers:
- name: myapp
image: myapp:latest
ports:
- containerPort: 9090
...
and this is the prometheus config
- job_name: 'kubernetes-pods'
scheme: http
metrics_path: /metrics
kubernetes_sd_configs:
- role: node
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
regex: myapp
action: keep
but I don't see any metrics in Prometheus or any mention of myapp
in Prometheus debug log. What am I missing?
My mistake. role had to be pod
instead of node
in the config.
I see that you didn't define - api_server: 'https://kubernetes'
. Make sure you define api-server in kubernetes_sd_config
. Prometheus auto discovers services via api-server.
kubernetes_sd_config
docs here