I have a reverse proxy from nginx docker image nginx:stable.

Dockerfile:

FROM nginx:stable

COPY main.conf /etc/nginx/conf.d/default.conf

COPY nginx.conf /etc/nginx/nginx.conf

COPY liveness /liveness

RUN chmod +x liveness/liveness_probe.sh

Reverse proxy sometimes randomly returns 502 bad gateway and in logs i can see:

2020/09/30 10:51:07 [error] 28#28: *67 connect() failed (110: Connection timed out) while connecting to upstream, client: xx.xx.xx.xx, server: , request: "POST xx HTTP/1.1", upstream: "xx", host: "xx"

main.conf:

server {
    listen      80;
    listen [::]:80;
    #Seconds to wait for timeout
    # fastcgi_read_timeout 300s;
    proxy_buffers           32 4m;
    proxy_busy_buffers_size     25m;
    proxy_buffer_size 512k;
    proxy_ignore_headers "Cache-Control" "Expires";
    proxy_max_temp_file_size 0;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Connection "";
    client_max_body_size        1024m;
    client_body_buffer_size     4m;
    proxy_connect_timeout 600;
    proxy_send_timeout 600;
    proxy_read_timeout 600;
    send_timeout 600;
    proxy_intercept_errors off;
    proxy_http_version 1.1;
    location ^~ /  {
        proxy_pass https://xx.com;    
}
}

nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  2048;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

The really weird thing is that if i use the same configuration in a vm with ubuntu 20.04 and install nginx package (without docker) the reverse proxy works without random 110 connection errors...

What could be wrong here ?

Everything is in aws - eks behind a alb ingress controller.

To discard ingress controller issues i also exposed with load balancer service and the issue remains, so discarded any alb ingress controller issues...

UPDATE:

Okay so another test i just did is in the same vm that the reverse proxy is working did a docker build and docker run, bind i t to another port and it wont return 502, it seems its a problem with the integration node where the docker container is scheduled.