Kubernetes dashboard nodeport not working

9/3/2019

I am trying to run kubernetes dashboard with NodePort. But this seems not to work I am following the following steps:

  1. Install kubernetes through kubeadm version 1.15.3
  2. Install flannel as pod network
  3. Install kubernetes dashboard with port type as NodePort

I get the following when I do kubctl logs ... kubernetes-dashboard

2019/09/03 01:22:31 Starting overwatch
2019/09/03 01:22:31 Using namespace: kubernetes-dashboard
2019/09/03 01:22:31 Using in-cluster config to connect to apiserver
2019/09/03 01:22:31 Using secret token for csrf signing
2019/09/03 01:22:31 Initializing csrf token from kubernetes-dashboard-csrf secret
2019/09/03 01:22:31 Empty token. Generating and storing in a secret kubernetes-dashboard-csrf
2019/09/03 01:22:31 Successful initial request to the apiserver, version: v1.15.3
2019/09/03 01:22:31 Generating JWE encryption key
2019/09/03 01:22:31 New synchronizer has been registered: kubernetes-dashboard-key-holder-kubernetes-dashboard. Starting
2019/09/03 01:22:31 Starting secret synchronizer for kubernetes-dashboard-key-holder in namespace kubernetes-dashboard
2019/09/03 01:22:32 Initializing JWE encryption key from synchronized object
2019/09/03 01:22:32 Creating in-cluster Sidecar client
2019/09/03 01:22:32 Auto-generating certificates
2019/09/03 01:22:32 Successfully created certificates
2019/09/03 01:22:32 Serving securely on HTTPS port: 8443
2019/09/03 01:22:32 Successful request to sidecar

Seems good to me, so far. Now to get the nodeport I do kubectl get svc --namespace kubernetes-dashboard

Now I get

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)         AGE
dashboard-metrics-scraper   ClusterIP   10.108.213.154   <none>        8000/TCP        11m
kubernetes-dashboard        NodePort    10.109.111.248   <none>        443:30206/TCP   11m

Now when I visit https://master-ip:30206 I just get a timeout, the log stays the same..

Service yaml:

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
  selector:
    k8s-app: kubernetes-dashboard

Deployment yaml:

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.0.0-beta4
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates
            - --namespace=kubernetes-dashboard
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
      volumes:
        - name: kubernetes-dashboard-certs
          secret:
            secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

Output when I apply the yaml:

:namespace/kubernetes-dashboard created
serviceaccount/kubernetes-dashboard created
service/kubernetes-dashboard created
secret/kubernetes-dashboard-certs created
secret/kubernetes-dashboard-csrf created
secret/kubernetes-dashboard-key-holder created
configmap/kubernetes-dashboard-settings created
role.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrole.rbac.authorization.k8s.io/kubernetes-dashboard created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
clusterrolebinding.rbac.authorization.k8s.io/kubernetes-dashboard created
deployment.apps/kubernetes-dashboard created
service/dashboard-metrics-scraper created
deployment.apps/dashboard-metrics-scraper created
-- ARR
kubernetes

2 Answers

9/3/2019

Have you created ClusterRole, ClusterRoleBinding resources for kubernetes-dashboard serviceaccount.? if not, Can you use below definition file to apply the changes .?

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/master/aio/deploy/recommended/05_dashboard-rbac.yaml
-- Subramanian Manickam
Source: StackOverflow

9/4/2019

I have finally been able to fix this problem and I took the following steps:

My iptables seemed to be messed up, I found this out when I did the following command on the master server:

curl {master-ip/e.g 192.168.1.11}:{Nodeport of kubernetes dashboard}

This command was working and I was getting dashboard back, the same curl gave me timeout on my client machine.

So I fixed this by doing the following command on the master node:

sudo iptables -A FORWARD -j ACCEPT

Now another issue arises, when you visit chrome will give invalid certificate error

To fix this the following needs to be done:

Create a file called san.cnf with the following content:

[ req ]
default_bits        = 2048
distinguished_name  = req_distinguished_name
req_extensions      = req_ext
prompt              = no
[ req_distinguished_name ]
countryName             = NL
stateOrProvinceName     = Noord Brabant
localityName            = Rosmalen
organizationName        = ARRServices
commonName              = k8s.dashboard.prod
[ req_ext ]
subjectAltName = @alt_names
[alt_names]
DNS.1   = {optional only do this if you reverse proxy}
IP.1    = {IP of the machine that is running the dashboard usually the master node}

# Create certificate by executing the following commands
sudo mkdir /certs    
sudo chmod 777 -R /certs
openssl req -out /certs/dashboard.csr -newkey rsa:2048 -nodes -keyout /certs/dashboard.key -config san.cnf
openssl x509 -req -sha256 -days 3650 -in /certs/dashboard.csr -signkey /certs/dashboard.key -out /certs/dashboard.crt -extensions req_ext -extfile san.cnf
sudo chmod 777 -R /certs

Now your kubernetes-dashboard.yml file should look like this:

apiVersion: v1
kind: Namespace
metadata:
  name: kubernetes-dashboard

---

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  type: NodePort
  ports:
    - port: 443
      targetPort: 8443
      nodePort: 31000
  selector:
    k8s-app: kubernetes-dashboard

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-certs
  namespace: kubernetes-dashboard
type: Opaque

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-csrf
  namespace: kubernetes-dashboard
type: Opaque
data:
  csrf: ""

---

apiVersion: v1
kind: Secret
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-key-holder
  namespace: kubernetes-dashboard
type: Opaque

---

kind: ConfigMap
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard-settings
  namespace: kubernetes-dashboard

---

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
rules:
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
  - apiGroups: [""]
    resources: ["secrets"]
    resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
    verbs: ["get", "update", "delete"]
    # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["kubernetes-dashboard-settings"]
    verbs: ["get", "update"]
    # Allow Dashboard to get metrics.
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["heapster", "dashboard-metrics-scraper"]
    verbs: ["proxy"]
  - apiGroups: [""]
    resources: ["services/proxy"]
    resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
    verbs: ["get"]

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
rules:
  # Allow Metrics Scraper to get metrics from the Metrics server
  - apiGroups: ["metrics.k8s.io"]
    resources: ["pods", "nodes"]
    verbs: ["get", "list", "watch"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard
subjects:
  - kind: ServiceAccount
    name: kubernetes-dashboard
    namespace: kubernetes-dashboard

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
        - name: kubernetes-dashboard
          image: kubernetesui/dashboard:v2.0.0-beta4
          imagePullPolicy: Always
          ports:
            - containerPort: 8443
              protocol: TCP
          args:
            - --auto-generate-certificates=false
            - --namespace=kubernetes-dashboard
            - --tls-cert-file=dashboard.crt
            - --tls-key-file=dashboard.key
            # Uncomment the following line to manually specify Kubernetes API server Host
            # If not specified, Dashboard will attempt to auto discover the API server and connect
            # to it. Uncomment only if the default does not work.
            # - --apiserver-host=http://my-address:port
          volumeMounts:
            - name: kubernetes-dashboard-certs
              mountPath: /certs
              # Create on-disk volume to store exec logs
            - mountPath: /tmp
              name: tmp-volume
          livenessProbe:
            httpGet:
              scheme: HTTPS
              path: /
              port: 8443
            initialDelaySeconds: 30
            timeoutSeconds: 30
      volumes:
        - name: kubernetes-dashboard-certs
          hostPath:
            path: /certs
          #secret:
          #  secretName: kubernetes-dashboard-certs
        - name: tmp-volume
          emptyDir: {}
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  ports:
    - port: 8000
      targetPort: 8000
  selector:
    k8s-app: dashboard-metrics-scraper

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: dashboard-metrics-scraper
  name: dashboard-metrics-scraper
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: dashboard-metrics-scraper
  template:
    metadata:
      labels:
        k8s-app: dashboard-metrics-scraper
    spec:
      containers:
        - name: dashboard-metrics-scraper
          image: kubernetesui/metrics-scraper:v1.0.1
          ports:
            - containerPort: 8000
              protocol: TCP
          livenessProbe:
            httpGet:
              scheme: HTTP
              path: /
              port: 8000
            initialDelaySeconds: 30
            timeoutSeconds: 30
          volumeMounts:
          - mountPath: /tmp
            name: tmp-volume
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      volumes:
        - name: tmp-volume
          emptyDir: {}

changes that I have made are the following: Service of kubernetes-dashboard will have:

NodePort instead of ClusterIP

  • Static nodePort 31000 instead of dynamic port
  • Args of kubernetes-dashboard deployment as follow:

    - --auto-generate-certificates=false
    - --namespace=kubernetes-dashboard
    - --tls-cert-file=dashboard.crt
    - --tls-key-file=dashboard.key
  • replaced secret and secretName from volumes kubernetes-dashboard-certs with hostPath: path: /certs # refers to the path where we generated the certs

    - name: kubernetes-dashboard-certs
      hostPath:
        path: /certs

Now execute the following command: 'kubectl apply -f kubernetes-dashboard.yml'

Now you will be able to visit it through {ip}:{nodePort}

  • The ip is usually the master node ip
  • The nodeport can be retrieved by doing:

    kubectl get svc --namespace kubernetes-dashboard

Most browser will give an error this looks ugly, to fix this do the following (Mac OSX, using chrome):

  • Open chrome and go to the ip and port of the dashboard e.g. "192.168.1.11:31000"
  • You will see that the certificate is not trusted, so download the certificate: go to developer tools => security => view certificate, and drag the certificate icon/logo to the desktop
  • Open keychain => go to keychains "System" => go to category "Certificates"
  • Drag the certificate to the list of certificates
  • Double click the certificate to open it, click on 'trust' - select at "when using
  • This certificate" the "always trust" option

Now you should be able to visit the dashboard next step is to create admin user this can be done as follow:

Create a file called serviceaccount.yml with the following content

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard

Create a file called rbac.yml with the following content:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard

Execute the following commands

kubectl apply -f serviceaccount.yml
kubectl apply -f rbac.yml

Get the token by executing this command

kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep admin-user | awk '{print $1}')

This is working with kubernetes 1.15.3, docker 18.09, kubernetes-dashboard 2.0.0beta4

Tested on: raspberry pi 4 as master node, but should also work with other devices.

-- ARR
Source: StackOverflow