For Traefik Ingress Controller in k3s disable TLS Verification

1/18/2020

I am using the default installation of k3s (release v1.17.0+k3s.1) and verified it is working correctly on my Raspberry Pi cluster.

After installing kubernetes-dashboard using the recommended instructions (https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) I confirmed this works, after I exposed it outside the cluster using the k3s traefik-based load balancer.

However, when I try exposing it using the traefik ingress controller I was unable to connect to kubernetes dashboard, with the browser showing a HTTP 500 error i.e. Internal Server Error. Below is my ingress yaml and kubernetes-dashboard pod logs showing the TLS certificate error. I believe the certificate error is due to traefik not knowing about (and trusting) the TLS certificate used by kubernetes dashboard.

a. k8s-dashboard-Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
    - host: k8s-services
      http:
        paths:
          - path: /k8s-dashboard
            backend:
              serviceName: kubernetes-dashboard
              servicePort: 443

Note. the domain "k8s-services" domain is defined in my pi cluster machines and browsing computers host file.

b. Kubernetes dashboard pod logs

...
2020/01/18 06:04:19 Auto-generating certificates
2020/01/18 06:04:19 Metric client health check failed: the server is currently unable to handle the request (get services dashboard-metrics-scraper). Retrying in 30 seconds.
2020/01/18 06:04:19 Successfully created certificates
2020/01/18 06:04:19 Serving securely on HTTPS port: 8443
2020/01/18 06:04:49 Successful request to sidecar
2020/01/18 06:23:28 http: TLS handshake error from 10.42.0.113:52698: remote error: tls: bad certificate
2020/01/18 06:24:33 http: TLS handshake error from 10.42.0.113:52886: remote error: tls: bad certificate
2020/01/18 06:24:41 http: TLS handshake error from 10.42.0.113:52908: remote error: tls: bad certificate
2020/01/18 06:24:47 http: TLS handshake error from 10.42.0.113:52926: remote error: tls: bad certificate
2020/01/18 06:24:53 http: TLS handshake error from 10.42.0.113:52948: remote error: tls: bad certificate
2020/01/18 06:25:37 http: TLS handshake error from 10.42.0.113:53076: remote error: tls: bad certificate
2020/01/18 06:25:41 http: TLS handshake error from 10.42.0.113:53090: remote error: tls: bad certificate
2020/01/18 06:25:44 http: TLS handshake error from 10.42.0.113:53102: remote error: tls: bad certificate
2020/01/18 06:31:22 http: TLS handshake error from 10.42.0.113:54144: remote error: tls: bad certificate
2020/01/18 06:31:29 http: TLS handshake error from 10.42.0.113:54158: remote error: tls: bad certificate
...

My issue seems very similiar to the following Stack Overflow issue: Kubernetes dashboard through Ingress

My first attempt to address this issue is to add the "ssl.insecureSkipVerify: 'true'" to the traefik.toml file in k3s. Unfortunately I could not find a global traefik.toml file (and traefik seems to be installed using a Helm Chart).

Please let me know ways of updating the "ssl.insecureSkipVerify" setting in traefik, or other alternate solutions.

-- Michael
k3s
kubernetes-dashboard
traefik-ingress

0 Answers