There are multiple apps hosted in my Rancher cluster and I continue to observe sporadic 504 Bad Gateway errors. The NGINX ingress controller log shows following errors:

These ingress controllers do not show the IP next to them and show following error:

[error] 314#314: *4 upstream timed out (110: Connection timed out) while connecting to upstream, client:

These log entries pertain to the 504 Bad Gateway error: controller.go:769] Service "SERVICENAMEREMOVED/ingress-d212988e26a4daf163485eaf33b959fa" does not have any active Endpoint.

I added proxy-connect-timeout and proxy-read-timeout with 120s as value in NGINX config. I then restarted the NGINX ingress controllers, some of which do not come up with an IP assigne.