Prometheus targets showing 403 for kubelet

10/16/2019

I was using prometheus for the monitoring of pod's cpu and network usage. but the metrics like cpu_usage_seconds are not coming in prometheus.

when i checked the the kubelet target's are down.

I'm using stable/prometheus-operator from helm:

Screenshot prometheus

-- user8177026
kubelet
kubernetes-helm
prometheus
prometheus-operator

1 Answer

10/16/2019

There are several ways to troubleshoot this:

  1. Check permissions, webhooks and kube controller. Details can be found here

  2. Check if your firewall rule is not blocking the connection (on a proper port).

  3. Prometheus needs read access to all cluster components in order to get the metrics. Check the cluster roles.

  4. Check the service endpoint discovery configurations in the config map.

  5. make sure you are using the latest stable version of the Prometheus.

Please let me know if that helped.

-- OhHiMark
Source: StackOverflow