I'm in the process of packaging an application to deploy and run on an on-prem Kubernetes cluster. Everything has been great so far, but I'm struggling on a component that is essentially a database. This component requires 3 master nodes, with 2 replicas. What's interesting is the master node itself counts as a replica, and needs to be configured as such (this is for sharding purposes). Since this is on-prem, I need to balance the master node and replica pairs across 3 physical hosts, with unique node and replica pairs running on each host.
This is an example of what the layout would look like, where N == a master node, and R == replica.
Host1 Host2 Host3
N1R1 N2R1 N3R1
N3R2 N1R2 N2R2I can generate the required config using StatefulSets and some helm magic with the init container ordinals, but I'm struggling to pin the N1R1 and N3R2 to Host 1, N2R1 and N1R2 instances to Host 2, and N3R1 and N2R2 instances to Host 3. I need to do this to ensure data integrity. Am I missing something with persistent volume controllers? Or some setting in StatefulSets that lets me access the underlying Kubernetes node to define which node and replica pairs run where?
What about Affinity and AntiAffinity? Can I leverage these to force a secondary replica instance to never be on the same node as it's master instance?
You are looking for Anti-Affinity. To avoid scheduling 2 Pods on the same Node you need to use topologyKey: "kubernetes.io/hostname" in the resource description.
Here is the link on the official documentation with the example.
And here is the link for deeper understanding how it works.