K
Q

Kubernetes Dashboard, CrashLoopBackOff, user not in namespace "kube-system"

2/26/2019

I am trying to run Kubernetes dashboard. I've followed the steps in the official wiki.

Listing kube-system pods, I see:

kubectl get pods -n kube-system
kubernetes-dashboard-head-7478c547df-8bmxf  0/1  CrashLoopBackOff  1  12s

Restarting this pod causes the same crash. In logs it appears to have crashed because:

kubectl logs -n kube-system kubernetes-dashboard-head-7478c547df-8bmxf  
2019/02/26 23:15:57 Starting overwatch
2019/02/26 23:15:57 Using namespace: kube-system
2019/02/26 23:15:57 Using in-cluster config to connect to apiserver
2019/02/26 23:15:57 Using secret token for csrf signing
2019/02/26 23:15:57 Initializing csrf token from kubernetes-dashboard-csrf secret
panic: secrets "kubernetes-dashboard-csrf" is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard-head" cannot get secrets in the namespace "kube-system": RBAC: role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal-head" not found

I don't fully understand how this can be. The serviceaccounts from this namespace container kubernetes-dashbaord-head:

kubectl get serviceaccounts -n kube-system
kubernetes-dashboard-head            1         8h

Since this serviceaccount (is this the same as "User"?) is listed in the serviceaccounts under namespace kube-system, why can't it get secrets from that namespace?

With recards to RBAC: role.rbac.authorization.k8s.io ... not found, is this related to this issue?

This is not on minikube.

-- Display name
kubernetes
rbac

Similar Questions

Is it possible to disable spectre/meltdown/related patches on coreos?
Token generated for GKE auth is missing privileges
Variable from kubernetes yaml causes: Could not resolve placeholder 'ACTUATOR_PASSWORD' in value "${ACTUATOR_PASSWORD}"
Kubernetes, simple SpringBoot app OOMKilled
Connecting to a Postgres DB running in K8s (Docker Desktop)
How does AKS handle the .env file in a container?
Azure AD B2C with EasyAuth for Kubernetes

2 Answers

2/28/2019

First of all, your case is not related to this issue. This is just similar error that you get.

You need to have appropriate Role and RoleBinding which you probably don't have, that's why you get this error. You can verify with below command:

kubectl get role,rolebinding -n kube-system |grep kubernetes-dashboard-minimal-head

If you look at latest deployment yaml file of Kubernetes dashboard from your installation guide, you will find below Role and RoleBinding resources.

# ------------------- Dashboard Role & Role Binding ------------------- #

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubernetes-dashboard-minimal-head
  namespace: kube-system
rules:
  # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]
...

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubernetes-dashboard-minimal-head
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard-minimal-head
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard-head
  namespace: kube-system

But I am not sure how you deployed your deployment, so I would suggest you just delete your deployment yaml file, via

kubectl delete -f <your-deployment-file.yaml>

then apply latest which includes Role and RoleBindings for Dashboard

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/kubernetes-dashboard-head.yaml

Then you can verify again via:

kubectl get role,rolebinding -n kube-system |grep kubernetes-dashboard-minimal-head

And check your pods of course. I hope it would be useful

-- coolinuxoid
Source: StackOverflow
10/4/2019

As you error suggests secrets "kubernetes-dashboard-csrf" is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard-head" cannot get secrets in the namespace "kube-system": RBAC: role.rbac.authorization.k8s.io "kubernetes-dashboard-minimal-head" not found, you are running k8s-dashboard pod under kubernetes-dashboard-head serviceaccount in kube-system namespace. And the service account does not have access to get/list the secret kubernetes-dashboard-csrf. Now to fix this problem you need to grant access to get that secret. You can do it in two ways: 1. allow the dashboard to run as cluster-admin role 2. just allow it to get that secret in that namespace. The second option is much bette from a security point of view. To do this you need to add/edit your clusterrole (probably it is clusterrole since you want to view/edit objects cluster wide) that is bound to this serviceaccount (kubernetes-dashboard-head) as the following:

- apiGroups:
  - ""
  resourceNames:
  - kubernetes-dashboard-csrf
  resources:
  - secrets
  verbs:
  - get
  - list
  - create
  - update
-- Munir Hossain
Source: StackOverflow