ENETUNREACH error on external calls

11/29/2017

In the past week, I started getting an unusual error while fetching external information (in this specific case, facebook):

ConnectionError: HTTPSConnectionPool(host='graph.facebook.com', port=443): Max retries exceeded with url: /v2.7/xxxxxx/xxxxx (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 101] ENETUNREACH',))

I'm using the GCE load balancer and a few times per day I get the 502 no gateway error. There is only one instance (for now) behind the load balancer.

I'm tempted to believe I'm loosing internet connectivity on this instance for a short time (maybe a few seconds, the load balancer needs to make a few calls to flag it as working).

Anyone got issues with connectivity dropping on a compute engine instance? Anyone has an idea how to debug it?

-- PierrePaul
google-cloud-platform
google-compute-engine
google-kubernetes-engine

1 Answer

12/1/2017

I faced the following problem working with Nginx together with HTTP Load Balancer experiencing sporadic 502 bad gateway and you might find it helpful. Basically you I had to tune the server in order to make sure that the connection was not dropped by a timeout due to a race condition.

The main issue is that "the default nginx keepalive_timeout is incompatible with the Google Cloud Platform HTTP(S) Load Balancer".

You must increase nginx’s 'keepalive_timeout', or risk intermittent and sporadic 502 Bad Gateway responses to POST requests.

# Tune nginx keepalives to work with the GCP HTTP(S) Load Balancer:
keepalive_timeout 650;
keepalive_requests 10000;

You can find an awesome article here that helped me before, that shows the root cause of this problem.

EDIT. Sometimes I had issues also with the NGINX Rate Limiting setting that was causing the HTTP health check to fail and to consider the instance unhealthy for a while, I don't know if it is related to your issue but it might be helpful for someone finding your question searching the 502 error.

-- GalloCedrone
Source: StackOverflow