K
Q

nginx-ingress return default backend on https request

12/4/2019

My AKS is accessible via a nginx-ingress. Everything works with https but since I use https nginx is not able to match any routes and use the default backend.

I'm using Kubernetes Version 1.15. I changed my domain to example.com and the IP to 51.000.000.128. The SSL certificate is signed by an external provider (digicert).

ingress-controller

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml

ingress-service

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-ingress
  namespace: ingress-nginx
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - kp-user'
spec:
  tls:
  - hosts:
    - example.com
    secretName: ssl-secret
  rules:
  - host: example.com
  - http:
      paths:
      - path: /app1(/|$)(.*)
        backend:
          serviceName: app1-service
          servicePort: 80
      - path: /app2(/|$)(.*)
        backend:
          serviceName: app2-service
          servicePort: 80

The Ingress is running:

$ kubectl -n ingress-nginx get ing
NAME            HOSTS           ADDRESS          PORTS     AGE
nginx-ingress   example.com     51.000.000.128   80, 443   43h

And the description of the Ingress:

$ kubectl describe ingress nginx-ingress --namespace=ingress-nginx
Name:             nginx-ingress
Namespace:        ingress-nginx
Address:          51.000.000.128
Default backend:  default-http-backend:80 (<none>)
TLS:
  ssl-secret terminates example.com
Rules:
  Host  Path  Backends
  ----  ----  --------
  *
        /app1(/|$)(.*)   app1-service:80 (10.244.1.10:80,10.244.2.11:80)
        /app2(/|$)(.*)   app2-service:80 (10.244.1.12:80,10.244.2.13:80)
Annotations:
  nginx.ingress.kubernetes.io/rewrite-target:        /$2
  nginx.ingress.kubernetes.io/ssl-passthrough:       true
  nginx.ingress.kubernetes.io/ssl-redirect:          false
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/auth-realm":"Authentication Required - kp-user","nginx.ingress.kubernetes.io/auth-secret":"basic-auth","nginx.ingress.kubernetes.io/auth-type":"basic","nginx.ingress.kubernetes.io/rewrite-target":"/$2","nginx.ingress.kubernetes.io/ssl-passthrough":"true","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"nginx-ingress","namespace":"ingress-nginx"},"spec":{"rules":[{"host":"example.com"},{"http":{"paths":[{"backend":{"serviceName":"app1-service","servicePort":80},"path":"/app1(/|$)(.*)"},{"backend":{"serviceName":"app2-service","servicePort":80},"path":"/app2(/|$)(.*)"}]}}],"tls":[{"hosts":["example.com"],"secretName":"ssl-secret"}]}}

  kubernetes.io/ingress.class:              nginx
  nginx.ingress.kubernetes.io/auth-realm:   Authentication Required - kp-user
  nginx.ingress.kubernetes.io/auth-secret:  basic-auth
  nginx.ingress.kubernetes.io/auth-type:    basic
Events:                                     <none>

Like I wrote in the beginning, unfortunately I get every time the 404 not found page from nginx, if I try to access on of the routes via https. The Secret is working because I'm able to see a valid certificate in my browser. The ingress is also working because with http I'm not facing any issues.

Issue

http://51.000.000.128/app1 => working
https://51.000.000.128/app1 => working but unsecure (browser use http)
example.com => not working (404 Not Found by nginx | default backend)

When I access the page via domain, it will be recognized by ingress-controller:

$ sudo kubectl logs nginx-ingress-controller-799dbf6fbd-bbxdp -n ingress-nginx

// https request
165.000.00.000 - - [05/Dec/2019:12:26:40 +0000] "GET /app1 HTTP/1.1" 308 177 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 500 0.000 [upstream-default-backend] [] - - - - 323deb61e1babdbca2006844d268b1ce
165.000.00.000 - - [05/Dec/2019:12:26:40 +0000] "GET /app1 HTTP/2.0" 404 179 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 306 0.001 [upstream-default-backend] [] 127.0.0.1:8181 190 0.000 404 d0cae28ba059531c78bffff38de2a84d
165.000.00.000 - - [05/Dec/2019:12:26:55 +0000] "GET /app1 HTTP/2.0" 404 179 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 44 0.000 [upstream-default-backend] [] 127.0.0.1:8181 190 0.000 404 db153c080e0116f8b730508b5ae0b0f3

// http request
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1 HTTP/1.1" 200 550 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 501 0.004 [ingress-nginx-app1-service-80] [] 10.244.1.10:80 1116 0.000 200 01beb82bb5173e7b0392660a9325c222
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1/styles.66c87fc4c5e0902762b4.css HTTP/1.1" 200 10401 "http://51.000.000.128/app1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 439 0.001 [ingress-nginx-app1-service-80] [] 10.244.2.11:80 70796 0.000 200 d367dfc0ae4db08c54dc6b0cb96e1f55
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1/polyfills-es2015.80abe0a50bdacb904507.js HTTP/1.1" 200 12933 "http://51.000.000.128/app1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 464 0.002 [ingress-nginx-app1-service-80] [] 10.244.1.10:80 37277 0.000 200 a2a4cd368a4badf1b6d2b202cf3958c5
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1/runtime-es2015.cd056c32d7e60bda4f6b.js HTTP/1.1" 200 1499 "http://51.000.000.128/app1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 462 0.000 [ingress-nginx-app1-service-80] [] 10.244.2.11:80 2728 0.000 200 f34c880d21f0172eeee3cc4f058c52a7
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1/main-es2015.2bb12b52c456e81e18a1.js HTTP/1.1" 200 164595 "http://51.000.000.128/app1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 459 0.029 [ingress-nginx-app1-service-80] [] 10.244.1.10:80 566666 0.028 200 7375f5092851e8407fe299c36c8a1b13
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1/18-es2015.b5bfc8f7102d1318aebc.js HTTP/1.1" 200 554 "http://51.000.000.128/app1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 426 0.002 [ingress-nginx-app1-service-80] [] 10.244.2.11:80 973 0.000 200 92e549e50e5ab6df5d456b31a8a34d8a
165.000.00.000 - - [05/Dec/2019:12:27:40 +0000] "GET /app1/assets/logo.svg HTTP/1.1" 200 2370 "http://51.000.000.128/app1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36" 443 0.003 [ingress-nginx-app1-service-80] [] 10.244.1.10:80 4717 0.000 200 c2503ed57519784af2988b70861302ec

From my understanding, the request by my domain works. For any reason, the ingress controller is not able to use/find the ingress via https. What I'm doing wrong.

-- Nico Schuck
azure-aks
kubernetes
nginx-ingress

Similar Questions

kubectl port-forward can be dangerous used in prod environment?
Odp.Net core support for kubernetes
uid : unable to do port forwarding: socat not found using helm
Istio VirtualService rewrite prefix works like exact match
Kubectl commands cannot be executed from another VM
How to update /etc/services for k8s pod / Docker container
Deployment affinity

1 Answer

12/5/2019

Problem 1:

It should be related to your nginx.ingress.kubernetes.io/ssl-passthrough: "true" configuration.

If you enabled ssl-passthrough, nginx-ingress will not try to decrypt the traffic for you. It would pass through the traffic straight to target service for decryption. In this way, path-based routing will not work because path is actually also encrypted. Also, none of other nginx ingress annotation will not due to the nature of basically not touching the request.

If that is not you want, you would like to remove the ssl-passthrough configuration and let nginx-ingress to terminate the HTTPS for you.

See following for more readings:

  1. https://docs.giantswarm.io/guides/advanced-ingress-configuration/#ssl-passthrough
  2. https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#ssl-passthrough

Problem 2:

In the ingress configuration. For spec => rules, there should be no - before http tag. Adding - will apply paths route to all hosts instead of just example.com route. There should be a conflict with the tls config that just apply tls to example.com hosts.

-- Fei
Source: StackOverflow