Narrow authorisation to list cluster scoped events in kubernetes

3/25/2019

I am trying to get heapster eventer to work on a cluster with RBAC enabled. Using the same roles that work for /heapster command does not seem to be sufficient.

On running the pod logs fill up with entries like this:

Failed to load events: events is forbidden: User "system:serviceaccount:kube-system:heapster" cannot list events at the cluster scope

Does anyone know the proper authorization for my heapster service account, short of admin rights?

Eventer deployment doc:

kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  labels:
    k8s-app: eventer
  name: eventer
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: eventer
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: eventer
    spec:
      serviceAccountName: heapster
      containers:
        - name: eventer
          image: k8s.gcr.io/heapster-amd64:v1.5.4
          imagePullPolicy: IfNotPresent
          command:
            - /eventer
            - --source=kubernetes:https://kubernetes.default
            - --sink=log
          resources:
            limits:
              cpu: 100m
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 200Mi
          terminationMessagePath: /dev/termination-log
      restartPolicy: Always
      terminationGracePeriodSeconds: 30

RBAC:

# Original: https://brookbach.com/2018/10/29/Heapster-on-Kubernetes-1.11.3.html
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: heapster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - nodes
      - namespaces
      - events
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - deployments
    verbs:
      - get
      - list
      - update
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get

Cluster role binding:

# Original: https://github.com/kubernetes-retired/heapster/blob/master/deploy/kube-config/rbac/heapster-rbac.yaml
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: heapster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: heapster
subjects:
  - kind: ServiceAccount
    name: heapster
    namespace: kube-system

Related question: How to propagate kubernetes events from a GKE cluster to google cloud log

-- Gudlaugur Egilsson
heapster
kubernetes
logging

1 Answer

3/25/2019

All of the above objects seem to be correct to me.

It's just a hunch but perhaps you created the Deployment first and then the ClusterRole and/or ClusterBindingRole and/or the ServiceAccount itself. Make sure you have these 3 first, then delete the current heapster Pods (or the Deployment, and wait for the Pod to terminate before recreating the Deployment).

(Create the ServiceAccount by kubectl create sa heapster -n kube-system)

Also, you can test if ServiceAccount can list the events by:

kubectl get ev --all-namespaces --as system:serviceaccount:kube-system:heapster
-- Janos Lenart
Source: StackOverflow