We have an existing Kubernetes cluster running that is using Istio. I was planning on adding a new Prometheus pod and can find plenty of blogs on how to do it. However, I noticed Istio already has a Prometheus service running in the Istio-System namespace.
My main goal is to get Grafana running with a few basic monitoring dashboards. Should I go ahead and use Istio's Prometheus service? What are the advantages/disadvantages of using Istio's Prometheus service over running my own?
i would suggested instead of installing Prometheus in Cluster better to use Azure log analytics.
Step 1 : Create log analytics in azure Step 2 : Install OMS agent in kubernetes cluster with acts a pipe line between cluster and Log analytics Step 3: logs will start moving to Log analytics from Cluster Step 4: create Log Alerts from Azure monitoring which will monitor the cluster health Step 5 : Add Log analytics in grafana data source and create Few dashboards
Follow below link to connect Log analytics with Grafana https://www.ciraltos.com/connect-grafana-to-azure-log-analytics/
Follow link to create log analytics: https://docs.microsoft.com/en-us/azure/container-service/kubernetes/container-service-kubernetes-oms
I'd suggest not sharing the existing istio prometheus, it's deployed in the istio-system namespace for a reason. It was deployed by and configured for istio.
If you really want to create a central shared prometheus service, use prometheus-operator and create a prometheus operator for istio. This is still going to be a lot of config effort to reintegrate your istio installation back into this new prometheus instance and is probably only worth it if you plan on scaling the number of clusters running this setup. 2 or 4 Prometheis is a manageable gap. 20 or 40 not so much.
Istio comes with Prometheus and a pre-configured version of Grafana including a Prometheus datasource and an Istio dashboard. You should be able to confirm that Prom and Grafana are already running in that namespace by running:
$ kubectl -n istio-system get svc prometheus
$ kubectl -n istio-system get svc grafana
(https://istio.io/docs/tasks/telemetry/metrics/using-istio-dashboard/)
They should be fully functional versions of both services, but in order to configure Prometheus for your custom jobs you'll need to find and update the ConfigMap deployed as part of Istio which includes prometheus.yaml.That leaves you open to losing those configuration options if you stop running Istio or if someone updates the configMap back to an old version for Istio reasons. The default image for Istio's Grafana is also an older one, version 5.2.3 according to the docs, so running your own would mean you can update to the newest version.
Running Prometheus twice shouldn't be a major issue especially if the two versions are focussed on different targets (and possibly run in different nodes though), so for example if your Prom is focussed on Node Exporter and Application Metrics, and the Istio Prom is only looking at Istio resources. That would keep it cleaner, and you can deploy your own Prometheus and other tools in a dedicated monitoring namespace. Here’s a blog and video I wrote about Prometheus deployment which covers namespaces and basic Prometheus/Node Exporter deployment.
Another option would be to use the Prometheus remote storage option and a remote, hosted version of Grafana. Metricfire (who I work for) provide remote storage and allow you to query that storage directly through a datasource in Grafana instead of targeting your local Prometheus. You can add remote storage details to any Prometheus config, including the Istio Prom if you like, and send it to Metricfire to host and create dashboards from (here’s a blog post I wrote about where to find prometheus.yaml for different deployment methods, if that helps). That would allow you to view metrics from both Proms side by side in the same dashboard.
You can also install the Istio dashboards there - you can find them in the Grafana site: (https://grafana.com/grafana/dashboards?search=istio). Having your metrics and the Istio metrics in the same location means you can also look at them side by side, and keep them for longer if you need to analyse any performance trends.