We're running a Kubernetes cluster on GKE, with an NGINX ingress controller (so that we can accept WS connections), and a network load balancer connected to it through a service to make it accessible from the internet.
Everything works perfectly fine most of the time. But we get "connection refused" at seemingly random intervals.
Looking at the logs, it seems as if the dropped requests never hit NGINX (NGINX doesn't log anything at all for these requests). We've tried recreating the external load balancer, the service and so on but without success. Not sure where to go from here or where the issue lies.
Does anyone recognise the problem? Or have any potential solution? Help greatly appreciated!
Can you check if all kube-proxy pods in your cluster are functioning correctly? Use the following command:
kubectl get --namespace=kube-system pod -l component=kube-proxy -o wideMaybe some of the kube-proxy is not functioning properly. Hence causing intermittent connection problem.
Have you tried creating a new cluster? Strange things might happen sometimes in the world of GKE