Getting ERR_TOO_MANY_REDIRECTS on Ingress with HTTPS web services

7/29/2020

I have a Rancher cluster (v2.4.5) running on custom nodes with the following configuration:

  • External machine (example.com):
    • Runs Rancher server on port 8443;
    • Runs NGINX with (among other unrelated stuff) the following basic configuration:
user  nginx;
  
worker_processes 4;
worker_rlimit_nofile 40000;
  
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
  
events {
  worker_connections 8192;
}
 
http {
  upstream rancher_servers {
    least_conn;
    server <MY_NODE_IP>:443 max_fails=3 fail_timeout=5s;
  }

  server {
    listen 443 ssl http2;
    server_name example.com service1.example.com service2.example.com;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;

    location / {
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-Proto $scheme;
      proxy_set_header X-Forwarded-Port $server_port;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_pass https://rancher_servers;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection $connection_upgrade;
      proxy_read_timeout 900s;
    }
  }
}
  • Internal machine (MY_NODE_IP):
    • Runs Rancher agent (etcd/control plane/worker)

Firewall rules are OK, I can deploy minor web-apps with stuff running on port 80 only and get redirected automatically to HTTPS. An example of YAML I'm using to deploy stuff is the following:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: www-deployment
  labels:
    app: www
spec:
  replicas: 1
  selector:
    matchLabels:
      app: www
  template:
    metadata:
      labels:
        app: www
    spec:
      containers:
        - name: www
          image: my-www-image

---
kind: Service
apiVersion: v1
metadata:
  name: www-service
spec:
  selector:
    app: www
  ports:
    - port: 80

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: service1.example.com
    http:
      paths:
        - path: /
          backend:
            serviceName: www-service
            servicePort: 80

The problem is when I try to deploy a service that runs on both ports 80 and 443 but, when requested on port 80, automatically redirects to port 443. When that's the case, if I specify the Ingress like below (with port 443), I get a Bad Gateway response not from the host machine NGINX. I can tell that because my host machine runs nginx/1.18.0 and the response comes from nginx/1.17.10.

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: service1.example.com
    http:
      paths:
        - path: /
          backend:
            serviceName: www-service
            servicePort: 443

But then, if I change the configuration above to servicePort: 80 I keep getting ERR_TOO_MANY_REDIRECTS, because it enters an infinite loop of redirecting from anything to https://anything.

Am I doing anything wrong here? How can I do a workaround to make these things work?

-- Gabriel Milan
kubernetes
kubernetes-ingress
nginx
nginx-ingress
rancher

1 Answer

7/30/2020

Found it out. Turns out that the only thing I needed to do was to tell the nginx-ingress-controller that I was expecting HTTPS connections. Final YAML for exposing the service is the following:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: www-ingress
  annotations:
    ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
    - host: service1.example.com
      http:
        paths:
          - path: /
            backend:
              serviceName: www-service
              servicePort: 443
-- Gabriel Milan
Source: StackOverflow