I'd like to access the Prometheus service run by Istio from another pod in my cluster. On my local development environment using Kind, I'm able to access it at http://prometheus.istio-system.svc.cluster.local:9090. I was hoping to do the same thing on GKE, so I spun up a cluster using the Istio addon in GKE:
gcloud beta container clusters create $GCLOUD_CLUSTER_NAME \
--addons=HorizontalPodAutoscaling,Istio \
--machine-type=n1-standard-2 \
--num-nodes=4 \
--cluster-version=latest --zone=$GCLOUD_CLUSTER_ZONE \
--enable-stackdriver-kubernetes --enable-ip-alias \
--enable-autorepair \
--scopes cloud-platform \
--preemptible
At first, I was confused because I didn't see a prometheus service in the istio-system namespace and then I learned that they renamed the prometheus service to promsd for Istio on GKE. I attempted to access Prometheus at http://promsd.istio-system.svc.cluster.local:9090/ but it didn't work. I tried to telnet to it from another pod to test connectivity but I got a Connection Refused:
root@shiny-wordcloud-69684cd88-lhxc8:/# telnet promsd.istio-system.svc.cluster.local 9090
Trying 10.0.1.215...
telnet: Unable to connect to remote host: Connection refusedI then saw that there are instructions for enabling Prometheus for Istio on GKE and was pretty confused since I though Prometheus came with Istio out of the box.
My question are:
I was over your case and follow this documentation:
a) You have to enable an istio gateway and virtual service to allow configuration at envoy proxy level and then allow connection to prometheus:
Running this command and could get the forwarding:
$ kubectl -n istio-system port-forward $(kubectl -n istio-system get pod -l app=promsd -o jsonpath='{.items[0].metadata.name}') 9090:9090
Forwarding from 127.0.0.1:9090 -> 9090It's working.
This is the documentation, that I followed:
And for your second question:
I found this link in the public docs from Google: Monitoring Prometheus
I hope it helps.
Regarding your first question I'd say that you must expose promsd service in order to be able to access it. Exposing this service means making it possible to see metrics from the cluster from the internet. So, be aware of this. To accomplish this you may execute similar command e.g.:
kubectl expose deployment promsd --type=LoadBalancer --name=promsd
For the second one, the purpose of using Prometheus depends on what kind of monitoring tool suit you best. And for accessing this metrics is you may use Stackdriver Monitoring or Graphana.