I have installed kubernetes metrics-server https://github.com/kubernetes-sigs/metrics-server It works fine, I can use "kubectl top pods" & "kubectl top nodes", Now I want to scrape the metrics to prometheus outside kubernetes cluster to be shown on Grafana, What I have tried are
What should I do?
Try these out and let me know.
1.First you should run kube-state-metrics which collects all kubernetes metrics .
2.Using pod annotations on the kube-state-metrics , expose metrics like
prometheus.io/scrape: 'true'
prometheus.io/port: 'port'
prometheus.io/path: '/metrics'
prometheus.io/scheme: 'http'This should expose your metrics from the pod level . To check this you can exec in to the pod and do a curl request on the port followed by the path
3.Now run a prometheus agent which can scrape metrics from this port and send it to the backend db server
4.Configure prometheus ds in your grafana agent and it should be done
You can use helm to install kube-prometheus-stack.
It will install Node Exporter, kube-state-metrics and prometheus operator along with other dependencies to allow you to monitor your cluster.
You can install it by using:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add stable https://charts.helm.sh/stable
helm repo update
helm install [RELEASE_NAME] prometheus-community/kube-prometheus-stack
Once it is installed you can visit your grafana URL and configure dashboard.
You can add prometheus as data source Configuration -> Data Source in grafana (prometheus URL should be set by default) and import kubernetes dashboard to show you summary metrics about containers running on Kubernetes nodes.