I have an app providing RESTFull api in google kubernetes cluster. In front of application i have an nginx working as a proxy_pass. The problem is that one request of few thousands (1000, 2000) has bad data in response (other users data). Analysing logs showed that request of the bad response doesn't come to the application at all. But it comes to nginx:

2019/05/08 13:48:03 [warn] 5#5: *28350 delaying request, excess: 0.664, by zone "one", client: 10.240.0.23, server: myportal.com, request: "GET /api/myresource?testId=10 HTTP/1.1"

In the same time there's no logs in the app for testId=10 (but there are for testId=9 and testId=11 when i make sequential test 1..1000)

Nginx configuration is almost default

limit_req_zone $binary_remote_addr zone=one:10m rate=4r/s;

server {
listen 80 default_server;
listen [::]:80 default_server;
server_name myportal.com;

if ($http_x_forwarded_proto = "http") {
    return 308 https://$server_name;
}

charset utf-8;

access_log on;
server_tokens off;

location /api {
    proxy_pass http://backend-service:8000;
    limit_req zone=one burst=10;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

There is no caching configured (or maybe it's on by default?).

Application is working in google kubernetes environement, so the request chain looks like this

(k8s ingress, nginx-service) -> nginx -> (k8s backend-service) -> backend

Backend app is written in spring and using jetty to run. Nginx version was updated from 1.13.X to 1.15.12 but both has the same issue.

I have no idea what and where should i check to find the cause of the problem.