GKE and prometheus monitoring

4/26/2020

i have a GKE cluster (1.15) with monitoring and logging enabled. until now we've used the metrics-server for metrics monitoring on stackdriver. for other custom metrics we've used the custom-metrics-adapter that scrape the metrics and export them to stackdriver using prometheus-to-sd.

i wanted to start and see if prometheus can give us other capabilities, such as aggregated metrics for HPA for example.

on GCP marketplace, there is an option to deploy 'prometheus + Grafana' on GKE cluster. this basically installed few components: prometheus server + node exporters + alertsmanager + grafana + kube-state-metrics).

what i'm trying to understand is if installing prometheus alongside the metrics-server is a kind of duplication for resources metric. i mean if the default resource monitoring for cpu/mem for nodes and workdloads is the metric-server and we already have this kind of metrics on stackdriver, and the default prometheus.yaml includes cpu/mem resources metrics from kubernetes API, will we have duplicated metrics of that information? (assuming we will add stackdriver-prometheus that will export all to stackdriver)

should the best practice is usually to disable the metrics-server and let prometheus to be the only monitoring solution on the cluster , while including the prometheus-k8s-adapter? or should we just disable these sections on the prometheus.yaml to not include these resources metrics?

also, does all prometheus metrics will be defined as external ones and will includes additional charge?

thanks!

-- clevi
google-kubernetes-engine
prometheus

1 Answer

4/26/2020

i wanted to start and see if prometheus can give us other capabilities, such as aggregated metrics for HPA for example.

Yes, it's possible, see for example the following articles:

what i'm trying to understand is if installing prometheus alongside the metrics-server is a kind of duplication for resources metric.

The answer is no. metrics-server is lightweight, short-term, in-memory and mainly used for Horizontal and Vertical Pod Autoscaler. You shouldn't use metrics-server when you need an accurate source of resource usage metrics.

should the best practice is usually to disable the metrics-server and let prometheus to be the only monitoring solution on the cluster , while including the prometheus-k8s-adapter?

It's a good option to use 'prometheus + Grafana + node-exporter + kube-state-metrics + cAdvisor (integrated into the kubelet) + alertmanager + prometheus adapter'

-- dmkvl
Source: StackOverflow