rewrite-target annotations breaks all ingress rules

12/14/2021

I have 3 ingress rules in a cluster that point to 3 different services/deployments.

I want each app, which is configured to listen on /tenant to respond to their respective sub path, so:

https://example.com/foo -> foo service https://example.com/bar -> bar service

and so on

The ingress rule for foo looks like this.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: foo-platform
    meta.helm.sh/release-namespace: aks-foo-namespace
    nginx.ingress.kubernetes.io/client-header-buffer-size: 64k
    nginx.ingress.kubernetes.io/client_body_buffer_size: 64k
    nginx.ingress.kubernetes.io/http2-max-field-size: 16k
    nginx.ingress.kubernetes.io/http2-max-header-size: 128k
    nginx.ingress.kubernetes.io/large-client-header-buffers: 8 64k
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
    nginx.ingress.kubernetes.io/proxy-buffers: 4 256k
    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: 256k
  labels:
    app.kubernetes.io/managed-by: Helm
  name: foo-ingress
  namespace: foo
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: foo-service
            port:
              number: 80
        path: /foo1/(.*)
        pathType: Prefix
  tls:
  - hosts:
    - example.com
    secretName: example-tls-ingress

Now, this currently works, but only if another, unrelated namespace is configured like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: bar-platform
    meta.helm.sh/release-namespace: aks-bar-namespace
    nginx.ingress.kubernetes.io/client-header-buffer-size: 64k
    nginx.ingress.kubernetes.io/client_body_buffer_size: 64k
    nginx.ingress.kubernetes.io/http2-max-field-size: 16k
    nginx.ingress.kubernetes.io/http2-max-header-size: 128k
    nginx.ingress.kubernetes.io/large-client-header-buffers: 8 64k
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
    nginx.ingress.kubernetes.io/proxy-buffers: 4 256k
    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: 256k
    nginx.ingress.kubernetes.io/rewrite-target: $1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: bar-ingress
  namespace: bar
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: bar-service
            port:
              number: 80
        path: /bar/(.*)
        pathType: Prefix
  tls:
  - hosts:
    - example.com
    secretName: bar-tls-ingress

Notice the rewrite-target annotation which appears to need to be there. Unfortunately, this means that https://example.com/bar doesn't respond correctly, and is configured on https://example.com/bar/bar

I looked at the nginx.conf for a working foo ingress, and noticed the location block is wildly different if the rewrite-target annotations is present. When it's broken, the location block is set to location ~* so something isn't right.

What's the right way to configure subpaths for nginx-ingress? How can I make all 3 apps respond on https://example.com/app-tenant-name

-- jaxxstorm
kubernetes
nginx
nginx-ingress
nginx-reverse-proxy

1 Answer

12/14/2021

ideally, this should not be the case, Nginx won't be crashing or so. i would suggest updating the path and re-write annotation a little bit and check

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    meta.helm.sh/release-name: bar-platform
    meta.helm.sh/release-namespace: aks-bar-namespace
    nginx.ingress.kubernetes.io/client-header-buffer-size: 64k
    nginx.ingress.kubernetes.io/client_body_buffer_size: 64k
    nginx.ingress.kubernetes.io/http2-max-field-size: 16k
    nginx.ingress.kubernetes.io/http2-max-header-size: 128k
    nginx.ingress.kubernetes.io/large-client-header-buffers: 8 64k
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-buffer-size: 128k
    nginx.ingress.kubernetes.io/proxy-buffers: 4 256k
    nginx.ingress.kubernetes.io/proxy-busy-buffers-size: 256k
    nginx.ingress.kubernetes.io/rewrite-target: $2
  labels:
    app.kubernetes.io/managed-by: Helm
  name: bar-ingress
  namespace: bar
spec:
  rules:
  - host: example.com
    http:
      paths:
      - backend:
          service:
            name: bar-service
            port:
              number: 80
        path: /bar(/|$)(.*)
        pathType: Prefix
  tls:
  - hosts:
    - example.com
    secretName: bar-tls-ingress

updated the nginx.ingress.kubernetes.io/rewrite-target: $2 & path changes for bar service.

-- Harsh Manvar
Source: StackOverflow