Nginx Ingress with OAuth2 authentication 404 page not found in Kubernetes

11/10/2020

following the previous question on Stack Overflow at this link, after successful authentication (at Github.com) i get 404 page not found on my browser.

The Ingress configuration below (used by nginx-ingress controller):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: nginx-ingress
  annotations:
     nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
     nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$request_uri"
spec:
  ingressClassName: nginx
  rules:
  - host: site.example.com
    http:
      paths:
      - path: /v1                                               
        backend:
          serviceName: web-service
          servicePort: 8080
      - path: /
        backend:
          serviceName: oauth2-proxy
          servicePort: 4180
  tls:
  - hosts:
    - site.example.com
    secretName: example-tls

$ kubectl get ing -n nginx-ingress
  NAME      CLASS   HOSTS              ADDRESS   PORTS     
  ingress   nginx   site.example.com             80, 443   

The node.js web application i'm trying to access to via oauth2 has been built with two paths (/ and /v1). Web application is behind Service web-service.

OAuth2 Github application configuration:

Homepage URL
https://site.example.com/

Authorization callback URL
https://site.example.com/oauth2/callback

OAuth2 deployment and service:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: nginx-ingress
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: oauth2-proxy
  template:
    metadata:
      labels:
        k8s-app: oauth2-proxy
    spec:
      containers:
      - args:
        - --provider=github
        - --email-domain=*
        - --upstream=file:///dev/null
        - --http-address=0.0.0.0:4180
        # Register a new application
        # https://github.com/settings/applications/new
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          value: 32066******52
        - name: OAUTH2_PROXY_CLIENT_SECRET
          value: ff2b0a***************9bd
        - name: OAUTH2_PROXY_COOKIE_SECRET
          value: deSF_t******03-HQ==
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        imagePullPolicy: Always
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP

apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: oauth2-proxy
  name: oauth2-proxy
  namespace: nginx-ingress
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    k8s-app: oauth2-proxy

Logs from oauth2-proxy container:

[2020/11/10 19:47:27] [logger.go:508] Error loading cookied session: cookie "_oauth2_proxy" not present, removing session
10.44.0.2:51854 - - [2020/11/10 19:47:27] site.example.com GET - "/" HTTP/1.1 "Mozilla/5.0
[2020/11/10 19:47:27] [logger.go:508] Error loading cookied session: cookie "_oauth2_proxy" not present, removing session
10.44.0.2:51858 - - [2020/11/10 19:47:27] site.example.com GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 ....
10.44.0.2:51864 - - [2020/11/10 19:47:28] site.example.com GET - "/oauth2/start?rd=%2F" HTTP/1.1 "Mozilla/5.0 ....
10.44.0.2:52004 - marco.***81@gmail.com [2020/11/10 19:48:33] [AuthSuccess] Authenticated via OAuth2: Session{email:marco.***81@gmail.com user:mafi81 PreferredUsername: token:true created:2020-11-10 19:48:32.494549621 +0000 UTC m=+137.822819581}
10.44.0.2:52004 - - [2020/11/10 19:48:32] site.example.com GET - "/oauth2/callback?code=da9c3af9d8f35728d2d1&state=e3280edf2430c507cd74f3d4655500c1%3A%2F" HTTP/1.1 "Mozilla/5.0 ...
10.44.0.2:52012 - marco.****81@gmail.com [2020/11/10 19:48:33] site.example.com GET - "/" HTTP/1.1 "Mozilla/5.0 ....
10.44.0.2:52014 - marco.****81@gmail.com [2020/11/10 19:48:33] site.example.com GET - "/favicon.ico" HTTP/1.1 "Mozilla/5.0 .... Chrome/86.0.4240.193 Safari/537.36" 404 19 0.000

Testing environment:

  • VirtualBox with kubeadm v1.19.3
  • NGINX Ingress controller Version=1.9.0.

I'm not still confident with paths configuration under Ingress resource. Any suggestion on how to go ahead with troubleshoot would be great.

UPDATE:

Following the Matt's answer, giving the right way to test the authentication, here is the new environment:

NGINX Ingress controller
  Release:       v0.41.2
  Build:         d8a93551e6e5798fc4af3eb910cef62ecddc8938
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.4

OAuth2 Pod
  image: quay.io/oauth2-proxy/oauth2-proxy

Ingress manifest:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  namespace: web
  annotations:
     nginx.ingress.kubernetes.io/auth-response-headers: Authorization
     nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.web.svc.cluster.local:4180/oauth2/auth
     nginx.ingress.kubernetes.io/auth-signin: https://site.example.com/oauth2/start?rd=$request_uri
     nginx.ingress.kubernetes.io/configuration-snippet: |
       auth_request_set $name_upstream_1 $upstream_cookie__oauth2_proxy_1;

       access_by_lua_block {
         if ngx.var.name_upstream_1 ~= "" then
           ngx.header["Set-Cookie"] = "_oauth2_proxy_1=" ..  ngx.var.name_upstream_1 .. ngx.var.auth_cookie:match("(; .*)")
         end
       }

spec:
  ingressClassName: nginx-oauth
  rules:
  - host: site.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: web-service
          servicePort: 8080

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: web

spec:
  ingressClassName: nginx-oauth

  rules:
  - host: site.example.com
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 4180
        path: /oauth2

  tls:
  - hosts:
    - site.example.com
    secretName: tls

Note that i had to change one annotation to get it working:

  • auth-url: http://oauth2- proxy.web.svc.cluster.local:4180/oauth2/auth (this solves resolution failure)
-- Marco_81
kubernetes
nginx
oauth-2.0

1 Answer

11/12/2020

According to oauth-proxy documentation you MUST use kubernetes/ingress-nginx.

Here you can read more about differences between nginxinc/kubernetes-ingress and kubernetes/ingress-nginx Ingress Controllers.

In oath2-proxy docs (mentioned earlier) you can find the following:

When you use ingress-nginx in Kubernetes, you MUST use kubernetes/ingress-nginx (which includes the Lua module) and the following configuration snippet for your Ingress. Variables set with auth_request_set are not set-able in plain nginx config when the location is processed via proxy_pass and then may only be processed by Lua. Note that nginxinc/kubernetes-ingress does not include the Lua module.

nginx.ingress.kubernetes.io/auth-response-headers: Authorization
nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$escaped_request_uri
nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
nginx.ingress.kubernetes.io/configuration-snippet: |
  auth_request_set $name_upstream_1 $upstream_cookie_name_1;

  access_by_lua_block {
    if ngx.var.name_upstream_1 ~= "" then
      ngx.header["Set-Cookie"] = "name_1=" .. ngx.var.name_upstream_1 .. ngx.var.auth_cookie:match("(; .*)")
    end
  }

So if we can trust documentation, your authentication won't work because you are using wrong nginx controller and you are missing annotations.

-- Matt
Source: StackOverflow