Prometheus / Grafana setup on kubernetes (gcloud)

11/14/2018

I wanted to setup some monitoring and tried this: https://github.com/giantswarm/kubernetes-prometheus

It sets up all the Pods and services but I get an error message:

Error from server (Forbidden): error when creating
"https://raw.githubusercontent.com/giantswarm/kubernetes-prometheus/master/manifests-all.yaml":
clusterroles.rbac.authorization.k8s.io "prometheus" is forbidden:
attempt to grant extra privileges: [PolicyRule{APIGroups:[""],
Resources:["nodes"], Verbs:["get"]} PolicyRule{APIGroups:[""],
Resources:["nodes"], Verbs:["list"]} PolicyRule{APIGroups:[""],
Resources:["nodes"], Verbs:["watch"]} PolicyRule{APIGroups:[""],
Resources:["nodes/proxy"], Verbs:["get"]} PolicyRule{APIGroups:[""],
Resources:["nodes/proxy"], Verbs:["list"]} PolicyRule{APIGroups:[""],
Resources:["nodes/proxy"], Verbs:["watch"]} PolicyRule{APIGroups:[""],
Resources:["services"], Verbs:["get"]} PolicyRule{APIGroups:[""],
Resources:["services"], Verbs:["list"]} PolicyRule{APIGroups:[""],
Resources:["services"], Verbs:["watch"]} PolicyRule{APIGroups:[""],
Resources:["endpoints"], Verbs:["get"]} PolicyRule{APIGroups:[""],
Resources:["endpoints"], Verbs:["list"]} PolicyRule{APIGroups:[""],
Resources:["endpoints"], Verbs:["watch"]} PolicyRule{APIGroups:[""],
Resources:["pods"], Verbs:["get"]} PolicyRule{APIGroups:[""],
Resources:["pods"], Verbs:["list"]} PolicyRule{APIGroups:[""],
Resources:["pods"], Verbs:["watch"]} PolicyRule{APIGroups:[""],
Resources:["configmaps"], Verbs:["get"]}
PolicyRule{NonResourceURLs:["/metrics"], Verbs:["get"]}]
user=&{xxx  [system:authenticated]
map[user-assertion.cloud.google.com:[xxx]]}
ownerrules=[PolicyRule{APIGroups:["authorization.k8s.io"],
Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"],
Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis"
"/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json"
"/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}
PolicyRule{NonResourceURLs:["/openapi"], Verbs:["get"]}
PolicyRule{NonResourceURLs:["/openapi/*"], Verbs:["get"]}
PolicyRule{NonResourceURLs:["/version/"], Verbs:["get"]}]
ruleResolutionErrors=[]

I would appreciate any help. I suppose I need to grant extra rights, but I dont know how. Cluster version is: 1.10.9-gke.5

Thank you

EDIT: With the changes @Rico proposed I am getting an slightly different error message:

Error from server (Forbidden): error when creating "manifests-all.yaml": clusterroles.rbac.authorization.k8s.io "prometheus" is forbidden: attempt to grant extra privileges: 
[PolicyRule{APIGroups:["*"], Resources:["*"], Verbs:["*"]} PolicyRule{NonResourceURLs:["/metrics"], Verbs:["get"]}] user=&{xxx  [system:authenticated] map[user-assertion.cloud.google.com:[xxx]]} ownerrules= 
[PolicyRule{APIGroups:["authorization.k8s.io"], Resources:["selfsubjectaccessreviews" "selfsubjectrulesreviews"], Verbs:["create"]} 
PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swagger-2.0.0.pb-v1" "/swagger.json" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]} PolicyRule{NonResourceURLs:["/openapi"], Verbs:["get"]} 
PolicyRule{NonResourceURLs:["/openapi/*"], Verbs:["get"]} 
PolicyRule{NonResourceURLs:["/version/"], Verbs:["get"]}] ruleResolutionErrors=[]
-- Alex Tbk
google-cloud-platform
kubernetes
prometheus

1 Answer

11/15/2018

You need to set up your prometheus ClusterRole to be more permissive. You can start with full permissions to see if it works initially and then work your way down:

cat <<EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: prometheus
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
EOF | kubectl apply -f -
-- Rico
Source: StackOverflow