I have one service which serves as an endpoint to upload documents into our system. We started doing some load tests and notices that we get some connection timeout errors for some of our files.

In my project we use:

1. Java 9
2. Spring Boot 2.0
3. Azure AKS
4. By default, we have ingress controllers in front of our services

Together with my teammates, we tried to collect some statistics and that is what we have:

1. If we upload 2000 documents with 5MB per one document in 128 threads we have the following errors: Error Status 999:194, Error Status 504:1
2. If we upload 2000 documents with 250KB per one document in 48 threads sometimes we have the following errors: Error Status 999:2

I have checked AKS, Ingress, and Spring Boot documentation and I do not see that it has any rate limits enabled by default. Moreover, I do not see any errors in the logs of my Java App, so it seems that everything happens somewhere in AKS and Ingress.

You can find our ingress config example:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ZZZ-ingress-dev-files-storage
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: files-storage
    nginx.ingress.kubernetes.io/whitelist-source-range: "whitelisted-ips"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 151m
spec:
  tls:
  - hosts:
    - files-storage.dev.ZZZ-env.net
    secretName: tls-files-storage.dev.ZZZ-env.net

  rules:
  - host: files-storage.dev.ZZZ-env.net
    http:
      paths:
      - path: /
        backend:
          serviceName: files-storage-svc
          servicePort: 80

And here you can find examples of logs from ingress controller:

14:30:28  2019/08/05 06:28:20 [warn] 43#43: *1177362 a client request body is buffered to a temporary file /tmp/client-body/0000082261, client: 91.185.25.194, server: file-storage.test.ZZZ-env.net, request: "POST /ZZZ/files HTTP/1.1", host: "file-storage.test.ZZZ-env.net"
14:30:28  2019/08/05 06:28:20 [warn] 43#43: *1177342 a client request body is buffered to a temporary file /tmp/client-body/0000082262, client: 91.185.25.194, server: file-storage.test.ZZZ-env.net, request: "POST /ZZZ/files HTTP/1.1", host: "file-storage.test.ZZZ-env.net"

Do you have any ideas what can be a reason of "rejected" request? Something in AKS or Ingress configs enabled by default?