I have a service configured as a LoadBalancer running on a five-node cluster running 1.9.11. The LoadBalancer sits in front of three pods running an ASP.NET Core web application (that in turn talks to a NATS message queue from which a listener retrieves messages and saves them to an Azure SQL database). All pods have resource request and limits set and everything is in a dedicated namespace.
I’m using a PowerShell script to cause the web application to generate a message for the NATS queue every 50 milliseconds. I can see from a couple of ways that the Loadbalancer is only sending traffic to one pod: firstly the CPU graphs in the k8s dashboard show no activity for two of the pods and secondly I’m tracing the Environment. MachineName from the web app right the way through to a field in the database and I can see that it’s only ever one MachineName. If I delete the pod that is receiving traffic a new pod immediately stars receiving traffic but it's still only that one pod out of three.
My understanding is that this isn’t how the LoadBalancer is intended to work, ie the LoadBalancer should send traffic to all pods. Is that right and if so any clues as to what I’m doing wrong? My service file is as follows:
apiVersion: v1
kind: Service
metadata:
name: megastore-web-service
spec:
selector:
app: megastore-web
ports:
- port: 80
type: LoadBalancerIt sounds to me like your load balancer is working correctly. When traffic comes into a LB the LB will automatically direct traffic to the first available node. The fact that you can shutdown your POD and traffic is rerouted is what would be expected.
This is a good article which helps explain how the LB works
To test this further, I would suggest you try opening a port on one of the PODs but not the others. Such as port 88 on POD2. Then connect using the loadbalancer:88 and see if the connection gets routed to the correct POD.