I am trying to get heapster eventer to work on a cluster with RBAC enabled. Using the same roles that work for /heapster command does not seem to be sufficient.
On running the pod logs fill up with entries like this:
Failed to load events: events is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list events at the cluster scope
Does anyone know the proper authorization for my heapster service account, short of admin rights?
Eventer deployment doc:
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
labels:
k8s-app: eventer
name: eventer
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
k8s-app: eventer
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
labels:
k8s-app: eventer
spec:
serviceAccountName: heapster
containers:
- name: eventer
image: k8s.gcr.io/heapster-amd64:v1.5.4
imagePullPolicy: IfNotPresent
command:
- /eventer
- --source=kubernetes:https://kubernetes.default
- --sink=log
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
terminationMessagePath: /dev/termination-log
restartPolicy: Always
terminationGracePeriodSeconds: 30
RBAC:
# Original: https://brookbach.com/2018/10/29/Heapster-on-Kubernetes-1.11.3.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: heapster
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- namespaces
- events
verbs:
- get
- list
- watch
- apiGroups:
- extensions
resources:
- deployments
verbs:
- get
- list
- update
- watch
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
Cluster role binding:
# Original: https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: heapster
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: heapster
subjects:
- kind: ServiceAccount
name: heapster
namespace: kube-system
Related question: How to propagate kubernetes events from a GKE cluster to google cloud log
All of the above objects seem to be correct to me.
It's just a hunch but perhaps you created the Deployment first and then the ClusterRole and/or ClusterBindingRole and/or the ServiceAccount itself. Make sure you have these 3 first, then delete the current heapster Pods (or the Deployment, and wait for the Pod to terminate before recreating the Deployment).
(Create the ServiceAccount by kubectl create sa heapster -n kube-system
)
Also, you can test if ServiceAccount can list the events by:
kubectl get ev --all-namespaces --as system:serviceaccount:kube-system:heapster