I am trying to scrape data from Istio envoy at 15090 port using prometheus.
My current setup is with istio 1.1.5 with standalone prometheus(not the one which comes attached with istio)
Envoy sidecars are attached to multiple pods in different namespaces and I am not sure how to scrape data on specific port in multiple istio-proxy containers
I tried using service monitor to scrape data from istio envoy and its not working.
Service monitor which I tried currently.
kind: ServiceMonitor
metadata:
annotations:
labels:
k8s-app: istio
name: envoy
namespace: monitoring
spec:
endpoints:
- interval: 5s
path: /metrics
port: http-envoy-prom
jobLabel: envoy
namespaceSelector:
matchNames:
- istio-system
selector:
matchLabels:
istio: mixer```
can somebody help, how to scrape data from port 15090 on multiple istio-proxy containers attached to multiple pods.Apart from ServiceMonitor you also need to create following scrape config for envoy proxies
# Scrape config for envoy stats
- job_name: 'envoy-stats'
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:15090
target_label: __address__
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod_name
metric_relabel_configs:
# Exclude some of the envoy metrics that have massive cardinality
# This list may need to be pruned further moving forward, as informed
# by performance and scalability testing.
- source_labels: [ cluster_name ]
regex: '(outbound|inbound|prometheus_stats).*'
action: drop
- source_labels: [ tcp_prefix ]
regex: '(outbound|inbound|prometheus_stats).*'
action: drop
- source_labels: [ listener_address ]
regex: '(.+)'
action: drop
- source_labels: [ http_conn_manager_listener_prefix ]
regex: '(.+)'
action: drop
- source_labels: [ http_conn_manager_prefix ]
regex: '(.+)'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_tls.*'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_tcp_downstream.*'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_http_(stats|admin).*'
action: drop
- source_labels: [ __name__ ]
regex: 'envoy_cluster_(lb|retry|bind|internal|max|original).*'
action: dropOr use this script