Prometheus Version - 6.7.0
In order to scrape only specific port for a service, I have used below configuration. In Prometheus the target service endpoint always goes in status
context deadline exceeded and Error says - Get http://<service-name>:8000/metrics:dial tcp <service-ip>:8000:i/o timeout
My Prometheus job configuration -
- job_name: '<service-name>'
scrape_interval: 10s
scrape_timeout: 20s
honor_labels: true
kubernetes_sd_configs:
- role: service
relabel_configs:
- source_labels: [__meta_kubernetes_service_label_app]
action: keep
regex: <app-name>
- source_labels: [__meta_kubernetes_service_name]
action: replace
target_label: kubernetes_name
- source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_scrape_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
Tried adding below tls_config but that didnt help -
tls_config:
insecure_skip_verify: true
Also I queried Prometheus to verify the scrape_duration_seconds
which is around 1 sec.
Is this a known issue? OR Do I need to follow any specific steps to get the detailed error logs.