I have a golang service (2 pods) and using a redis cluster (3 masters and 1 slave for each master) deployed in kubernetes, all the redis nodes are hidden behind a cluster ip in kubernetes. While load testing for my golang service I found that all the redis requests are going to only one node (Not sure why this is happening, according to the documentation replicas will also be used to read data from redis and as we have 2 pods on service running at all times we should be able to read data from 2 replicas).
Context for golang service
Golang service is opening just one connection at the start of the application and it is using that for getting data from redis.
Golang service is opening just one connection at the start of the application and it is using that for getting data from redis.
This is the reason why. When your application opens a connection, it is of course routed to one of the replicas. Until you keep the connection open you will use the same replica for all your requests. This has nothing to do with how kubernetes services work.
One easy solution would be to open a new connection everytime you need a redis client, but this comes with a performance penalty of course.
Another option would be to open a pool of connections on the client side and use all of them. Every connection should end up in a different replica and this way you should spread the load a little better.