K
Q

Unable to load kubernetes dashboard after successful oauth2

1/4/2022

I have deployment my kubernetes cluster v1.23.1 with kubeadm and configured it with the keycloak identity provider for authentication.

API server configuration for keycloak IDP

...
    - --oidc-issuer-url=https://kubemaster:8443/auth/realms/local
    - --oidc-client-id=gatekeeper
    - --oidc-username-claim=name
    - --oidc-groups-claim=groups
    - --oidc-ca-file=/etc/kubernetes/ssl/kubemaster.crt
...

I am running oauth2-proxy in another pod which authenticates with keycloak idp and provides the tokens (id_token). Here is my oauth2-proxy deployment with the service to expose it.

---

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
        - name: oauth2-proxy
          image: quay.io/oauth2-proxy/oauth2-proxy:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 4180
              protocol: TCP
          args:
          - --provider=oidc
          - --email-domain=*
          - --http-address=0.0.0.0:4180
          env:
          - name: OAUTH2_PROXY_OIDC_ISSUER_URL
            value: https://192.168.122.54:8443/auth/realms/local
          - name: OAUTH2_PROXY_REDIRECT_URL
            value: https://kubernetes-dashboard.localdev.me:8081/oauth2/callback
          - name: OAUTH2_PROXY_CLIENT_ID
            value: gatekeeper
          - name: OAUTH2_PROXY_CLIENT_SECRET
            value: jZzvJ0wCDDwltV3tAf0SXSbVoKXM1RqV
          - name: OAUTH2_PROXY_COOKIE_SECRET
            value: kgKUT3IMmESA81VWXvRpYIYwMSo1xndwIogUks6IS00=
          - name: OAUTH2_PROXY_UPSTREAM
            value: https://kubernetes-dashboard
          - name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
            value: "true"
          - name: OAUTH2_PROXY_COOKIE_DOMAIN
            value: .localdev.me
          - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
            value: "true"
          - name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
            value: "true"
          - name: OAUTH2_PROXY_SSL_UPSTREAM_INSECURE_SKIP_VERIFY
            value: "true"
          - name: OAUTH2_PROXY_OIDC_EMAIL_CLAIM
            value: email
          - name: OAUTH2_PROXY_GROUPS_CLAIM
            value: groups
          - name: OAUTH2_PROXY_ALLOWED_GROUPS
            value: developers

---

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  #type: NodePort
  ports:
    - port: 4180
      targetPort: 4180
      name: http
  selector:
    k8s-app: oauth2-proxy

I have nginx ingress deployed in front to route the request to oauth2-proxy as shown below.

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: oauth-proxy
    namespace: kubernetes-dashboard
    annotations:
      nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: kubernetes-dashboard.localdev.me
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: oauth2-proxy
            port:
              number: 4180
  tls:
  - hosts:
    - kubernetes-dashboard.localdev.me
    secretName: kubernetes-dashboard-ingress-tls

I am using port forward from localhost to ingress resource on port 443.

kubectl port-forward --namespace=ingress-nginx service/ingress-nginx-controller 8081:443

The issue currently i am facing is i am able to launch the url - https://kubernetes-dashboard.localdev.me:8081/ which routes to oauth2 proxy page from there i am able to launch the keycloak realm login page which i can authenticate. But once the keycloak authentication is successfully done i am not able to load the upstream which is kubernetes-dashboard service with the id_token. Instead i am getting a 404 not found for https://kubernetes-dashboard.localdev.me:8081/

oauth2-proxy Log details

[2022/01/04 19:06:24] [oauthproxy.go:866] No valid authentication in request. Initiating login.
192.168.1.169:33826 - 48490760d6e0fc519e13d3158878112a - - [2022/01/04 19:06:24] kubernetes-dashboard.localdev.me:8081 GET - "/" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 403 8033 0.000
[2022/01/04 19:06:25] [oauthproxy.go:866] No valid authentication in request. Initiating login.
192.168.1.169:33826 - a87fbdc228026d99e6fc9f29d9ffc6df - - [2022/01/04 19:06:25] kubernetes-dashboard.localdev.me:8081 GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 403 8044 0.000
192.168.1.169:33826 - 3c052e49f9f77e89c561f3d0bef47fed - - [2022/01/04 19:06:43] kubernetes-dashboard.localdev.me:8081 GET - "/oauth2/start?rd=%2F" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 302 356 0.000
192.168.1.169:33826 - a16b9b1b22ec03e58b9b930caa325f5b - alice@stack.com [2022/01/04 19:06:55] [AuthSuccess] Authenticated via OAuth2: Session{email:alice@stack.com user:52ed0190-4d9c-4749-90f4-37e5896bdc42 PreferredUsername:alice token:true id_token:true created:2022-01-04 19:06:55.77587057 +0000 UTC m=+290.895007232 expires:2022-01-04 19:11:55.775499536 +0000 UTC m=+590.894636200 refresh_token:true groups:[developers]}
192.168.1.169:33826 - a16b9b1b22ec03e58b9b930caa325f5b - - [2022/01/04 19:06:55] kubernetes-dashboard.localdev.me:8081 GET - "/oauth2/callback?state=HBDoNuX3mudlwQrjdEkxYP9yjgHn5_mqOXk8T-G21dg%3A%2F&session_state=e928739a-c94d-49ba-80a9-a1f83a936bf1&code=9a592559-236c-4896-86b4-ba8f657821e6.e928739a-c94d-49ba-80a9-a1f83a936bf1.b7b62a18-269b-480a-b5bd-ad3c16f94394" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 302 24 0.030
192.168.1.169:33826 - 9288833c2f8e82f2ad94c6260bcf68c7 - alice@stack.com [2022/01/04 19:06:55] kubernetes-dashboard.localdev.me:8081 GET - "/" HTTP/1.1 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 404 19 0.000

Ingress controller logs

127.0.0.1 - - [04/Jan/2022:19:02:26 +0000] "GET / HTTP/2.0" 404 19 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 2842 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 19 0.000 404 d19c25394498799f5024567ddd1fafcd
127.0.0.1 - - [04/Jan/2022:19:06:24 +0000] "GET / HTTP/2.0" 403 8033 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 297 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 8060 0.000 403 48490760d6e0fc519e13d3158878112a
127.0.0.1 - - [04/Jan/2022:19:06:25 +0000] "GET /favicon.ico HTTP/2.0" 403 8044 "https://kubernetes-dashboard.localdev.me:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 102 0.003 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 8071 0.004 403 a87fbdc228026d99e6fc9f29d9ffc6df
127.0.0.1 - - [04/Jan/2022:19:06:43 +0000] "GET /oauth2/start?rd=%2F HTTP/2.0" 302 356 "https://kubernetes-dashboard.localdev.me:8081/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 32 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 356 0.000 302 3c052e49f9f77e89c561f3d0bef47fed
127.0.0.1 - - [04/Jan/2022:19:06:55 +0000] "GET /oauth2/callback?state=HBDoNuX3mudlwQrjdEkxYP9yjgHn5_mqOXk8T-G21dg%3A%2F&session_state=e928739a-c94d-49ba-80a9-a1f83a936bf1&code=9a592559-236c-4896-86b4-ba8f657821e6.e928739a-c94d-49ba-80a9-a1f83a936bf1.b7b62a18-269b-480a-b5bd-ad3c16f94394 HTTP/2.0" 302 24 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 351 0.031 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 24 0.028 302 a16b9b1b22ec03e58b9b930caa325f5b
127.0.0.1 - - [04/Jan/2022:19:06:55 +0000] "GET / HTTP/2.0" 404 19 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:95.0) Gecko/20100101 Firefox/95.0" 2827 0.001 [kubernetes-dashboard-oauth2-proxy-4180] [] 192.168.1.177:4180 19 0.004 404 9288833c2f8e82f2ad94c6260bcf68c7

Please suggest on how i can route the authenticated session with id_token to kubernetes dashboard service for loading the dashboard.

-- sb9
kubernetes
kubernetes-dashboard
oauth2-proxy

Similar Questions

How to configure endpoints and esp to publish gRPC with enabled ServerReflection feature
How to enable VT-X/AMD-v on Google Cloud Platform VM's?
Helm jenkins/latest unable to run docker command
"Running on Docker with Consul or Eureka" ?
How to distribute requests to pods on a Kubernetes service

1 Answer

1/14/2022

I was finally able to resolve my issue with some updates to my yaml definition files.

Assuming you have a kubernetes cluster installed v1.23.1 with kubeadm on Ubuntu 20.04 and setup networking with flannel networking --pod-network-cidr=10.244.0.0/16. Also you have the keycloak oidc service setup (image - quay.io/keycloak/keycloak:16.1.0). Here are the updated yml definition files which helped me to resolve this issue.

Ingress controller applied -

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.0/deploy/static/provider/cloud/deploy.yaml

kubernetes dashboard applied -

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yaml

oauth2-proxy.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
      - args:
        - --provider=oidc
        - --email-domain=*
        - --http-address=0.0.0.0:4180
        env:
        - name: OAUTH2_PROXY_OIDC_ISSUER_URL
          value: https://kubemaster:8443/auth/realms/local
        - name: OAUTH2_PROXY_REDIRECT_URL
          value: https://<FQDN>/oauth2/callback
        - name: OAUTH2_PROXY_CLIENT_ID
          value: gatekeeper
        - name: OAUTH2_PROXY_CLIENT_SECRET
          value: jZzvJ0wCDDwltV3tAf0SXSbVoKXM1RqV
        - name: OAUTH2_PROXY_COOKIE_SECRET
          value: kgKUT3IMmESA81VWXvRpYIYwMSo1xndwIogUks6IS00=
        - name: OAUTH2_PROXY_UPSTREAM
          value: https://kubernetes-dashboard
        - name: OAUTH2_PROXY_SSL_INSECURE_SKIP_VERIFY
          value: "true"
        - name: OAUTH2_PROXY_INSECURE_OIDC_ALLOW_UNVERIFIED_EMAIL
          value: "true"
        - name: OAUTH2_PROXY_PASS_AUTHORIZATION_HEADER
          value: "true"
        - name: OAUTH2_PROXY_SSL_UPSTREAM_INSECURE_SKIP_VERIFY
          value: "true"
        - name: OAUTH2_PROXY_OIDC_EMAIL_CLAIM
          value: email
        - name: OAUTH2_PROXY_GROUPS_CLAIM
          value: groups
        - name: OAUTH2_PROXY_ALLOWED_GROUPS
          value: developers
        - name: OAUTH2_PROXY_SKIP_PROVIDER_BUTTON
          value: "true"
        - name: OAUTH2_PROXY_SET_AUTHORIZATION_HEADER
          value: "true"
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        imagePullPolicy: Always
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP

dashboard-ingress.yml 

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/auth-response-headers: "authorization"
  name: external-auth-oauth2
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: __INGRESS_HOST__
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443
---

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: kubernetes-dashboard
spec:
  ingressClassName: nginx
  rules:
  - host: __INGRESS_HOST__
    http:
      paths:
      - path: /oauth2
        pathType: Prefix
        backend:
          service:
            name: oauth2-proxy
            port:
              number: 4180
  tls:
  - hosts:
    - __INGRESS_HOST__
    secretName: __INGRESS_SECRET__

The issue with keycloak oidc provider its unable to set the Authroization header for which we need to manaully add the set authorization header in the oauth2-proxy defintion and also this header need to be appended in the response from the ingress to the upstream kubernetes dashboard by setting annotation - nginx.ingress.kubernetes.io/auth-response-headers: "authorization"

Also adding a reference for kubernetes dashboard authentication using github oauth2 which i was following to resolve this issue.

https://kubernetes.github.io/ingress-nginx/examples/auth/oauth-external-auth/#example-oauth2-proxy-kubernetes-dashboard
-- sb9
Source: StackOverflow