So i have 3 name spaces when i deployed prometheus on kubernetes i see the error in the logs. it is unable to monitor all the name spaces.
Error :
\"system:serviceaccount:development:default\" cannot list endpoints at the cluster scope" level=error ts=2018-06-28T21:22:07.390161824Z caller=main.go:216 component=k8s_client_runtime err="github.com/prometheus/prometheus/discovery/kubernetes/kubernetes.go:268: Failed to list *v1.Endpoints: endpoints is forbidden: User \"system:serviceaccount:devops:default\" cannot list endpoints at the cluster scope"
You'd better use a service account to access the kubernetes, and give the sa special privilidge that the prometheus needed. like the following:
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: kube-systemPresumes that you deploy prometheus in the kube-system namespace. Also you need specify the sa like ' serviceAccount: prometheus' in your prometheus deployment file .