Kubernetes NGINX ingress rewrite-target annotation breaking

6/23/2020

I am running kubernetes v1.16 under docker desktop for windows. I have installed the nginx-ingress controller v1.7.9 using helm. I have update my hosts file to have the following entry

127.0.0.1 application.local

I have a backend service named hedgehog-service.

The following ingress definition correctly forwards request to the backend.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ml-zoo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: application.local
    http:
      paths:
      - path: /hedgehog/
        backend:
          serviceName: hedgehog-service
          servicePort: 80

curl application.local/hedgehog works as expected and hits the backend service.

However in order to correctly use the backend service I need to rewrite the target removing the url prefix /hedgehog. Hence I have the following ingress definition

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ml-zoo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: application.local
    http:
      paths:
      - path: /hedgehog(/|$)(.*)
        backend:
          serviceName: hedgehog-service
          servicePort: 80

As indicated here: https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target

Now when I curl application.local/hedgehog/test the ingress controller does not communicate with the backend but according to the logs attempts to open a file

2020/06/23 12:46:48 [error] 708#708: *792 open() "/etc/nginx/html/hedgehog/test" failed (2: No such file or directory), client: 192.168.65.3, server: application.local, request: "GET /hedgehog/test HTTP/1.1", host: "application.local"
192.168.65.3 - - [23/Jun/2020:12:46:48 +0000] "GET /hedgehog/test HTTP/1.1" 404 153 "-" "curl/7.65.3" "-"

Here is the content of etc/nginx/conf.d/default-ml-zoo-ingress

# configuration for default/ml-zoo-ingress

upstream default-ml-zoo-ingress-application.local-hedgehog-service-80 {
        zone default-ml-zoo-ingress-application.local-hedgehog-service-80 256k;
        random two least_conn;

        server 10.1.0.48:80 max_fails=1 fail_timeout=10s max_conns=0;
}

server {
        listen 80;
        server_tokens on;
        server_name application.local;

        location /hedgehog(/|$)(.*) {

                proxy_http_version 1.1;
                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://default-ml-zoo-ingress-application.local-hedgehog-service-80;

        }
}

Does anyone know why my URLs are not getting rewritten and the requsts delivered to the backend service?

Thanks in advance!

-- Joe
docker-for-windows
kubernetes
kubernetes-ingress
nginx
nginx-ingress

1 Answer

6/23/2020

OK having played around with this for hours, once I had written the question my next google turned up an answer.

I installed nginx using helm with the following repo stable/nginx-ingress. However according to this issue https://github.com/kubernetes/ingress-nginx/issues/5756 that is in fact a legacy repository. I uninstalled my controller and changed the repository to ingress-nginx

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

and everything appears to be working as expected. Still not sure why the previous controller installation failed but I can get back to work :)

EDIT: For the aid of others that might end up here - in hindsight I am wondering if the reinstallation simply meant that I deleted and recreated my ingress which might have solved the original problem. In other words make sure you try recreating the ingress before reinstalling the ingress controller with helm.

-- Joe
Source: StackOverflow