I have a GKE cluster with this elasticseach logging solution installed
https://console.cloud.google.com/marketplace/details/google/elastic-gke-logging
And prometheus-operator installed by helm inside the same cluster.
I would like configure a grafana dashboard for visualize metrics of my elasticsearch.
I read that elastic application from gke has the elastic_exporter installed... https://github.com/GoogleCloudPlatform/click-to-deploy/blob/master/k8s/elastic-gke-logging/README.md
But if I go to my Prometheus panel I don't see any metric about elasticsearch. I try install another elastic_exporter, but nothing.
I miss something? I forget something? Do you need to configure prometheus to read from the elastic_exporter?
I see the metrics when I do port-forwarding of the elastic_exporter, but I don't see the metrics inside prometheus panel.
# HELP elasticsearch_breakers_estimated_size_bytes Estimated size in bytes of breaker
# TYPE elasticsearch_breakers_estimated_size_bytes gauge
elasticsearch_breakers_estimated_size_bytes{breaker="accounting",cluster="elastic-gke-logging-1-cluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="10.50.2.54",name="elastic-gke-logging-1-elasticsearch-0"} 4.6637464e+07
elasticsearch_breakers_estimated_size_bytes{breaker="fielddata",cluster="elastic-gke-logging-1-cluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="10.50.2.54",name="elastic-gke-logging-1-elasticsearch-0"} 0
elasticsearch_breakers_estimated_size_bytes{breaker="in_flight_requests",cluster="elastic-gke-logging-1-cluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="10.50.2.54",name="elastic-gke-logging-1-elasticsearch-0"} 0
elasticsearch_breakers_estimated_size_bytes{breaker="parent",cluster="elastic-gke-logging-1-cluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="10.50.2.54",name="elastic-gke-logging-1-elasticsearch-0"} 4.6637464e+07
elasticsearch_breakers_estimated_size_bytes{breaker="request",cluster="elastic-gke-logging-1-cluster",es_client_node="true",es_data_node="true",es_ingest_node="true",es_master_node="true",host="10.50.2.54",name="elastic-gke-logging-1-elasticsearch-0"} 0
# HELP elasticsearch_breakers_limit_size_bytes Limit size in bytes for breaker
# TYPE elasticsearch_breakers_limit_size_bytes gauge
Thank you
You are probably missing ServiceMonitor, this should work:
k apply -f -<<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
annotations:
labels:
release: prom
name: elasticsearch
spec:
endpoints:
- port: metrics
selector:
matchLabels:
app: es-exporter
EOFYour elasticsearch service must define metrics and have lable app: es-exporter, similar to this:
apiVersion: v1
kind: Service
metadata:
labels:
app: es-exporter
component: elasticsearch
name: elasticsearch
spec:
ports:
- name: transport
port: 9200
protocol: TCP
targetPort: 9200
- name: metrics
port: 9108
protocol: TCP
targetPort: 9108
selector:
component: elasticsearch
type: ClusterIPAfter that you should find metrics in Prometheus, to confirm that you can always use Status -> Targets tab in Prometheus.