K
Q

nginx ingress 504 timeout with running multi pod

11/13/2019

I am running 2 pod(replicas) of particular deployment on Kubernetes with nginx ingress. Service using web socket also.

Out of 2 pod I have deleted one pod so it starts creating again while 1 was in a ready state. In between this, I tried to open the URL and got an error 504 gateway timeout.

As per my understanding traffic has to divert to Ready state pod from Kubernetes service. Am I missing something please let me know?

Thanks in advance.

Here is my ingress if any mistake

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: core-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: core-prod
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/secure-backends: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/websocket-services: core
    nginx.org/websocket-services: core
spec:
  tls:
  - hosts:
    - app.wotnot.io
    secretName: core-prod
  rules:
  - host: example.io
    http:
      paths:
      - backend:
          serviceName: core
          servicePort: 80
-- Harsh Manvar
docker
google-kubernetes-engine
kubernetes
nginx
nginx-ingress

Similar Questions

certificate is valid for ingress.local, not gitlab.mydomain
Nginx routing ignores anything after path rule
Kubernetes + Redis: The antiforgery token could not be decrypted
Kubectl Java client returns exit code 3 when using exec

1 Answer

11/13/2019

Services do not guarantee 100% uptime, especially if there are only 2 pods. Depending on the timing of your request, one of a number of possible outcomes is occurring.

  1. You try to open the URL before the pod is marked as notReady. What happens, in this case, is your service forwards the request to your pod which is about to terminate. Since the pod is about to terminate and the webserver is shutting down, the pod is no longer able to respond so nginx responds with 504. It is also possible that a session is already started with this pod and it is interrupted because of the sigterm.

  2. You send a request once the second pod is in terminating state. Your primary pod is being overworked from handling 100% of the requests so a response does not come fast enough so nginx returns an error.

In any scenario, your best option is to check the nginx ingress container logs to see why 504 is being returned so you can further debug this.

Note that, as mentioned just above, services do only include pods marked as Ready, however, this does not guarantee that 100% of requests will always be served correctly. Any time a pod is taken down for any reason, there is always a chance that a 5xx error is returned. Having a greater number of pods will reduce the odds of an error being returned but it will rarely completely eliminate the odds.

-- Patrick W
Source: StackOverflow