Kubernetes service does not balance the load fairily under load

5/7/2020

We are having issues when we run an import in our system. He have a pod with 6 replicas that calls a service that is backed by a 5-replica pod. Nevertheless, our metrics indicate that, under load, that only one of the 5 pods is getting the request. We are using a ClusterIP service to "route" the calls.

We found this, more specifically this post but it is not clear if the problems he is reporting are only for long living connections.

We disabled the connection pool and we are still facing this unbalanced behavior under load.

We are running out of alternatives, so this is my question: Is it a known behavior (limitation?) with k8s services that we are having? It is documented somewhere?

PS: this service is only reachable from inside the cluster. PS2: Service definition

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2019-05-16T16:29:46Z"
  name: my-service
  namespace: my-ns
  .....
spec:
  clusterIP: <MyIp>
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: <my-selector>
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}
-- JSBach
kubernetes

1 Answer

5/14/2020

I found out the answer. I deployed an app that would call an endpoint through the service and would get back the node that answered. I noticed that during a time frame around 30s, only the same node would answer, then it would switch to another node for another time frame and then back. It seems weird, but seems to be the behavior of the service load balancing. At the end of the day, you will have a balanced load if you have a consistent amount of requests. Our request profile was "bursts" of requests in a short time, so this is the reason we got unbalanced loads.

-- JSBach
Source: StackOverflow