I'm having trouble getting kubernetes to allow access to the the dashboard. I am using dex/keycloack and am able to login successfully but then I receive an Unauthorized message from the UI. The browser console shows:
Couldn't get available api versions from server: Unauthorized\n","status":500
The kubernetes dashboard logs shows multiple errors, I have metrics-server installed but not heapster so I'm not as worried about the second one:
2019/08/23 12:15:11 Getting application global configuration
2019/08/23 12:15:11 Application configuration {"serverTime":1566562511915}
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Incoming HTTP/1.1 GET /api/v1/settings/global request from 192.168.2.12:38700: {}
2019/08/23 12:15:12 Cannot find settings config map: Unauthorized
2019/08/23 12:15:12 Cannot restore settings config map: Unauthorized
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Outcoming response to 192.168.2.12:38700 with 200 status code
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Incoming HTTP/1.1 GET /api/v1/systembanner request from 192.168.2.12:38700: {}
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Outcoming response to 192.168.2.12:38700 with 200 status code
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Incoming HTTP/1.1 GET /api/v1/login/status request from 192.168.2.12:38700: {}
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Outcoming response to 192.168.2.12:38700 with 200 status code
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Incoming HTTP/1.1 GET /api/v1/rbac/status request from 192.168.2.12:38700: {}
2019/08/23 12:15:12 Couldn't get available api versions from server: Unauthorized
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Outcoming response to 192.168.2.12:38700 with 500 status code
2019/08/23 12:15:12 [2019-08-23T12:15:12Z] Incoming HTTP/1.1 GET /api/v1/overview/default?filterBy=&itemsPerPage=10&name=&page=1&sortBy=d,creationTimestamp request from 192.168.2.12:38700: {}
2019/08/23 12:15:12 Getting config category
2019/08/23 12:15:12 Non-critical error occurred during resource retrieval: Unauthorized
2019/08/23 11:47:25 Metric client health check failed: the server could not find the requested resource (get services heapster). Retrying in 30 seconds.
I also see this in the keycloack logs which seems like it might be the issue but I haven't been able to determine why it would happen.
1.5665671075671058e+09 warn unable to parse the access token, using id token only {"error": "malformed JWS, only 1 segments"}
I'm using the following service account, role and role-binding but I've also tried binding the k8s-app: kubernetes-dashboard
account to the cluster-admin
role with no luck.
# ------------------- Dashboard Service Account ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
namespace: kube-system
---
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
rules:
# Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["create"]
# Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["create"]
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubernetes-dashboard-minimal
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: kube-system
---
This turns out to be a case of me not reading the documentation. I missed the section where I had to add these to the kube-api server.
--oidc-issuer-url=https://dex.k8s.example.com/dex: External Dex endpoint
--oidc-client-id=loginapp: ID for our Login Application
--oidc-username-claim=name: Map to nameAttr Dex configuration. This will be used by Kubernetes RBAC to authorize users based on their name.
--oidc-groups-claim=groups: This will be used by Kubernetes RBAC to authorize users based on their groups.