Specific requests being sent to general path on Nginx Ingress

8/26/2019

I have a Flask Python APP that has many routes. This app (container) is served in many different deployments, one for each plan type. Example: we have one deployment called esg-enterprise to process the enterprise plan, another esg-professional for professional and so on. Finally, we have another pod just to serve our frontend application with authentication and the models it needs. All this is the same container.

As you can see in the Ingress file bellow, we have the backend rules to route the traffic to specific services. The problem is that, most of the specific requests like /task/connections/update/advanced/ or /task/connections/update/advanced/ are being sent to the root path / that should only serve the front-end (which works since they use the same container). The problem is that those specific requests are very massive causing the front end API to become unavailable. The specific services run on stronger nodes so they can handle this load, while the front-end api runs on weaker nodes. When I run kubectl get hpa I can see that most of the load (replicas) stays on the API. I even had to increase the max replicas to try to mitigate the issue. I've seen in the logs that some requests are being sent to the specific routes as they should, but the majority is not being sent.

All I wanted to do is to prevent / route to receive those specific requests.

enter image description here

Our ingress looks like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/proxy-body-size: 150m
    nginx.ingress.kubernetes.io/proxy-connect-timeout: "1200"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1200"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1200"
    nginx.ingress.kubernetes.io/upstream-fail-timeout: "1200"
  name: ingress-nginx
  namespace: default

spec:
  tls:
  - hosts:
    - app.ourdomain.com
    - api.ourdomain.com
    secretName: ourdomain-certificate
  rules:
  - host: app.ourdomain.com
    http:
      paths:
      - backend:
          serviceName: frontend
          servicePort: 80
        path: /

  - host: api.ourdomain.com
    http:
      paths:
      - backend:
          serviceName: api
          servicePort: 8000
        path: /
      - backend:
          serviceName: esg
          servicePort: 8000
        path: /cron/
      - backend:
          serviceName: esg-bigquery-migration
          servicePort: 8000
        path: /cron/big-query/
      - backend:
          serviceName: esg
          servicePort: 8000
        path: /task/
      - backend:
          serviceName: esg-trial
          servicePort: 8000
        path: /task/connections/update/trial/
      - backend:
          serviceName: esg-advanced
          servicePort: 8000
        path: /task/connections/update/advanced/
      - backend:
          serviceName: esg-professional
          servicePort: 8000
        path: /task/connections/update/professional/
      - backend:
          serviceName: esg-enterprise
          servicePort: 8000
        path: /task/connections/update/enterprise/
-- Mauricio
kubernetes
nginx-ingress

1 Answer

8/28/2019

From documentation:

Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.

In some scenarios the exposed URL in the backend service differs from the specified path in the Ingress rule. Without a rewrite any request will return 404. To circumvent this you can set the annotation ingress.kubernetes.io/rewrite-target to the path expected by the service. Please, refer to documentation to learn more about this. Also, you should add following line to your annotations:

nginx.ingress.kubernetes.io/rewrite-target: /$1
-- muscat
Source: StackOverflow