I was experimenting load balancing our service by having two replicas on a cluster of two nodes.
Then I noticed the following:
Yesterday when I checked the pods, it looked like this:
pod-jq5vr 4/4 Running 0 2m 10.4.1.5 node-vvmb
pod-qbs69 4/4 Running 0 2m 10.4.0.10 node-jskq
This morning:
pod-hvjs8 4/4 Running 0 17h 10.4.1.6 node-vvmb
pod-jq5vr 4/4 Running 0 18h 10.4.1.5 node-vvmb
There must have been node recreation going on between yesterday and this morning, but the pod are on the same node now.
My questions:
You can use the pod anti-affinity feature to tell the scheduler that you don't want the pods in the same service to run on the same node.
The kubernetes documentation also has an example showing how to configure zookeper with anti-affinity for high availability.