K
Q

Kubernetes nginx ingress + oauth2 external auth timing out

5/22/2019

I am attempting to protect a service's status page with an oauth2_proxy, using Azure AD as the external auth provider. Currently if I browse to the public url of the app (https://sub.domain.com/service/hangfire) I got a 504 gateway timeout, where it should be directing me to authenticate.

I had been mostly following this guide for reference: https://msazure.club/protect-kubernetes-webapps-with-azure-active-directory-aad-authentication/

If I disable the annotations that direct the authentication, I can get to the public status page without a problem. If I browse to https://sub.domain.com/oauth2, I get a prompt to authenticate with my provider, which I would expect. I am not sure where the issue lies in the ingress config but I was unable to find any similar cases to this online, stackoverflow or otherwise.

In this case, everything (oauth deployment, service, and ingress rules) lives in a 'dev' namespace except the actual ingress deployment, which lives in its own namespace. I don't suspect this makes a difference, but SSL termination is handled by a gateway outside the cluster.

oauth2 deployment:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: oauth2-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - name: oauth2-proxy
        image: quay.io/pusher/oauth2_proxy:v3.2.0
        imagePullPolicy: IfNotPresent
        args:
        - --provider=azure
        - --email-domain=domain.com
        - --upstream=http://servicename
        - --http-address=0.0.0.0:4180
        - --azure-tenant=id
        - --client-id=id
        - --client-secret=number
        env:
         - name: OAUTH2_PROXY_COOKIE_SECRET
           value: secret
        ports:
         - containerPort: 4180
           protocol : TCP
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
spec:
  ports:
  - name: http
    port: 4180
    protocol: TCP
    targetPort: 4180
  selector:
    app: oauth2-proxy

Ingress rules:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-ingress1
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-url: https://sub.domain.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: https://sub.domain.com/oauth2/start?rd=$https://sub.domain.com/service/hangfire"
spec:
  rules:
  - host: sub.domain.com       
    http:
      paths:
      - path: /service/hangfire
        backend:
          serviceName: service
          servicePort: 80                    
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: service-oauth2-proxy
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: sub.domain.com      
    http:
      paths:
      - path: /oauth2
        backend:
          serviceName: oauth2-proxy
          servicePort: 4180

I am getting 504 errors when I browse to the url but I do not see any errors in the ingress pods.

-- stevenmiller
azure-active-directory
azure-aks
kubernetes
nginx-ingress
oauth-2.0

Similar Questions

how to setup basic rabbitmq on kubernetes
Allow AWS RDS connection from an Azure K8S pods
How to include a dictionary into a k8s_raw data field
How to validate that a Pod is running privileged
How to reload nginx ingress controller

3 Answers

5/23/2019

My setup is similar to 4c74356b41's

oauth2-proxy deployment

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - args:
        - --azure-tenant=TENANT-GUID
        - --email-domain=company.com
        - --http-address=0.0.0.0:4180
        - --provider=azure
        - --upstream=file:///dev/null
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              key: client-id
              name: oauth2-proxy
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              key: client-secret
              name: oauth2-proxy
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              key: cookie-secret
              name: oauth2-proxy
        image: quay.io/pusher/oauth2_proxy:v3.1.0
        name: oauth2-proxy

oauth2-proxy service

apiVersion: v1
kind: Service
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
  selector:
    app: oauth2-proxy
  type: ClusterIP

oauth2-proxy ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
spec:
  rules:
  - host: myapp.hostname.net
    http:
      paths:
      - backend:
          serviceName: oauth2-proxy
          servicePort: 80
        path: /oauth2

oauth2-proxy configuration

apiVersion: v1
kind: Secret
metadata:
  labels:
    app: oauth2-proxy
  name: oauth2-proxy
  namespace: monitoring
data:
# Values below are fake
  client-id: AAD_CLIENT_ID
  client-secret: AAD_CLIENT_SECRET
  cookie-secret: COOKIE_SECRET

Application using AAD Ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/auth-signin: https://$host/oauth2/start?rd=$request_uri
    nginx.ingress.kubernetes.io/auth-url: https://$host/oauth2/auth
  labels:
    app: myapp
  name: myapp
  namespace: monitoring
spec:
  rules:
  - host: myapp.hostname.net
    http:
      paths:
      - backend:
          serviceName: myapp
          servicePort: 80
        path: /
  tls:
  - hosts:
    - myapp.hostname.net

An additional step that needs to be done is to add the redirect URI to the AAD App registration. Navigate to your AAD App Registration in the Azure portal > Authentication > Add https://myapp.hostname.net/oauth2/callback to Redirect URIs > Save

-- Sean Boczulak
Source: StackOverflow
5/22/2019

This is what I've been doing with my oAuth proxy for Azure AD:

  annotations:
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"

And I've been using this oAuth proxy:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: oauth2-proxy
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: oauth2-proxy
  template:
    metadata:
      labels:
        app: oauth2-proxy
    spec:
      containers:
      - env:
          - name: OAUTH2_PROXY_PROVIDER
            value: azure
          - name: OAUTH2_PROXY_AZURE_TENANT
            value: xxx
          - name: OAUTH2_PROXY_CLIENT_ID
            value: yyy
          - name: OAUTH2_PROXY_CLIENT_SECRET
            value: zzz
          - name: OAUTH2_PROXY_COOKIE_SECRET
            value: anyrandomstring
          - name: OAUTH2_PROXY_HTTP_ADDRESS
            value: "0.0.0.0:4180"
          - name: OAUTH2_PROXY_UPSTREAM
            value: "http://where_to_redirect_to:443"
        image: machinedata/oauth2_proxy:latest
        imagePullPolicy: IfNotPresent
        name: oauth2-proxy
        ports:
        - containerPort: 4180
          protocol: TCP
-- 4c74356b41
Source: StackOverflow
5/23/2019

I ended up finding the resolution here: https://github.com/helm/charts/issues/5958

I had to use the internal service address for the auth-url, which I had not seen mentioned anywhere else.

nginx.ingress.kubernetes.io/auth-url: http://oauth2-proxy.development.svc.cluster.local:4180/oauth2/auth
-- stevenmiller
Source: StackOverflow