Context deadline exceeded for Prometheus service endpoint

12/13/2018

Prometheus Version - 6.7.0

In order to scrape only specific port for a service, I have used below configuration. In Prometheus the target service endpoint always goes in status

context deadline exceeded  and Error says - Get http://<service-name>:8000/metrics:dial tcp <service-ip>:8000:i/o timeout

My Prometheus job configuration -

- job_name: '<service-name>'
          scrape_interval: 10s
          scrape_timeout: 20s
          honor_labels: true
          kubernetes_sd_configs:
            - role: service
          relabel_configs:
            - source_labels: [__meta_kubernetes_service_label_app]
              action: keep
              regex: <app-name>
            - source_labels: [__meta_kubernetes_service_name]
              action: replace
              target_label: kubernetes_name
            - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_scrape_port]
              action: replace
              regex: ([^:]+)(?::\d+)?;(\d+)
              replacement: $1:$2
              target_label: __address__

Tried adding below tls_config but that didnt help -

tls_config:
  insecure_skip_verify: true

Also I queried Prometheus to verify the scrape_duration_seconds which is around 1 sec.

Is this a known issue? OR Do I need to follow any specific steps to get the detailed error logs.

-- Darshan Deshmukh
kubernetes
prometheus
prometheus-alertmanager

0 Answers