I have a Kubernetes v1.9.3 (no OpenShift) cluster I'd like to manage with ManageIQ (gaprindashvili-3 running as a Docker container).

I prepared the k8s cluster to interact with ManageIQ following these instructions. Notice that I performed the steps listed in the last section only (Prepare cluster for use with ManageIQ), as the previous ones were for setting up a k8s cluster and I already had a running one.

I successfully added the k8s container provider to ManageIQ, but the dashboard reports nothing: 0 nodes, 0 pods, 0 services, etc..., while I do have nodes, services and running pods on the cluster. I looked at the content of /var/log/evm.log of ManageIQ and found this error:

[----] E, [2018-06-21T10:06:40.397410 #13333:6bc9e80] ERROR – : [KubeException]: events is forbidden: User “system:serviceaccount:management-infra:management-admin” cannot list events at the cluster scope: clusterrole.rbac.authorization.k8s.io “cluster-reader” not found Method:[block in method_missing]

So the ClusterRole cluster-reader was not defined in the cluster. I double checked with kubectl get clusterrole cluster-reader and it confirmed that cluster-reader was missing.

As a solution, I tried to create cluster-reader manually. I could not find any reference of it in the k8s doc, while it is mentioned in the OpenShift docs. So I looked at how cluster-reader was defined in OpenShift v3.9. Its definition changes across different OpenShift versions, I picked 3.9 as it is based on k8s v1.9 which is the one I'm using. So here's what I found in the OpenShift 3.9 doc:

Name:       cluster-reader
Labels:     <none>
Annotations:    authorization.openshift.io/system-only=true
        rbac.authorization.kubernetes.io/autoupdate=true
PolicyRule:
  Resources                         Non-Resource URLs   Resource Names  Verbs
  ---------                         -----------------   --------------  -----
                                               [*]         []           [get]
  apiservices.apiregistration.k8s.io           []          []           [get list watch]
  apiservices.apiregistration.k8s.io/status    []          []           [get list watch]
  appliedclusterresourcequotas                 []          []           [get list watch]

I wrote the following yaml definition to create an equivalent ClusterRole in my cluster:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: 
  name: cluster-reader
rules:
- apiGroups: ["apiregistration"]
  resources: ["apiservices.apiregistration.k8s.io", "apiservices.apiregistration.k8s.io/status"]
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["*"]
  verbs: ["get"]

I didn't include appliedclusterresourcequotas among the monitored resources because it's my understanding that is an OpenShift-only resource (but I may be mistaken).

I deleted the old k8s container provider on ManageIQ and created a new one after having created cluster-reader, but nothing changed, the dashboard still displays nothing (0 nodes, 0 pods, etc...). I looked at the content of /var/log/evm.log in ManageIQ and this time these errors were reported:

[----] E, [2018-06-22T11:15:39.041903 #2942:7e5e1e0] ERROR -- : MIQ(ManageIQ::Providers::Kubernetes::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [kubernetes-01] as [] Event Monitor Thread aborted because [events is forbidden: User "system:serviceaccount:management-infra:management-admin" cannot list events at the cluster scope]
[----] E, [2018-06-22T11:15:39.042455 #2942:7e5e1e0] ERROR -- : [KubeException]: events is forbidden: User "system:serviceaccount:management-infra:management-admin" cannot list events at the cluster scope  Method:[block in method_missing]

So what am I doing wrong? How can I fix this problem? If it can be of any use, here you can find the whole .yaml file I'm using to set up the k8s cluster to interact with ManageIQ (all the required namespaces, service accounts, cluster role bindings are present as well).