I’m setting a RKE cluster in an EC2 AWS instances, but I have a problem trying to set up a nginx ingress controller sometimes I got error when try to access it. the architecture I have is this:

The instance #1 it just a nginx server that perform a load balancer in each node, The # 2 and # 3 are a RKE node both has those roles: - controlplane - worker - etcd

I have deployed two services/deployments. I trying to setup a nginx ingress controller to redirect the traffic to each service according to the path, but sometimes I just got 504 Gateway Time-out and others one load correctly. using hey to make a small load test I see that almost the 50% got the 504 error. Status code distribution: [200] 102 responses [504] 98 responses

Debugging the nginx-ingress controller I see that one of them seems not reach the service, I think for that reason sometimes I got 504 error but I don’t know why.

2020/01/27 01:40:31 [error] 1767#1767: *128496 upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.0.1.163, server: <host>, request: "GET /nginx HTTP/1.1", upstream: "http://10.42.1.4:80/", host: “<Host>"

The kubernetes configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
name: system-deployment
labels:
app: system
spec:
replicas: 1
selector:
matchLabels:
app: system
template:
metadata:
labels:
app: system
spec:
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-deployment
labels:
app: inventory
spec:
replicas: 1
selector:
matchLabels:
app: inventory
template:
metadata:
labels:
app: inventory
spec:
containers:
- name: inventory-container
image: dockersamples/101-tutorial
ports:
- containerPort: 80
---

apiVersion: v1
kind: Service
metadata:
name: system-service
spec:
selector:
app: system
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: inventory-service
spec:
selector:
app: inventory
ports:
- protocol: TCP
port: 80
targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: root-ingress
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
rules:
- host: <host>
http:
paths:
- path: /nginx
backend:
serviceName: system-service
servicePort: 80
- path: /
backend:
serviceName: inventory-service
servicePort: 80

My theory is that ingress-controller can’t reach the service in the other node for that I got the 504 Error, but As far as I know a service can accessed by any node in the cluster. someone knows what could happens here?

Thanks,