Nginx load balancer under high load (502 error)

1/12/2020

Main issue: We are using the "Nginx ingress controller" for balance traffic to "unicorn" workers. For checking application healthy, K8s send every 30s request to "http://myHost/health/check.json" and use one unicorn worker for that. While the workers are busy, for example, 3 busy from 3 available k8s think that the pod is unhealthy and restart the pod.

Following issue: While we are using Nginx retries, the Nginx ingress controller is getting a high load of traffic and response 502 errors of request that tries to get a response. The life of request is the timeout we set.

Notes:

  1. For now, we cant change the workers form unicorn to something else.
  2. We are using the "Lua" module in Nginx.
  3. Our ingress controller is mutch important then Nginx sidecar in fact that he serves more than one service.
  4. The flow is [Traffic] -> [Nginx ingress controller] -> [Nginx sidecar] -> [Unicorn workers]

Our questions is:

  1. How to protect on unicorn/pod to keepalive?
  2. What is the most reliable way to sleep request in Nginx?
  3. What is the secure way to limit connection?
  4. How to use retries in Nginx and Nginx ingress controller and don't get a 502 error code?
-- David Vaknin
kubernetes
load-balancing
lua
nginx
unicorn

0 Answers