Imagine that we have a cluster with 10 nodes and we want to launch a replica of an application on 3 of these nodes, how to do it?
I was thinking about it and I came up with the below solutions but each has a problem:
Label those 3 nodes and use that label in the application with replicas=3 but we might end up with two replicas on one node (for High Availability I want to prevent this).
Label those 3 nodes and use daemonset and use that label in daemonset but then we will have 7 pods in pending mode.
What is the solution to this problem?
Label those 3 nodes and use that label in the application with replicas=3 but we might end up with two replicas on one node (for High Availability I want to prevent this).
You may use podAntiAffinity rules, or topologySpreadConstraints, to avoid 2 Pods starting on the same node
Label those 3 nodes and use daemonset and use that label in daemonset but then we will have 7 pods in pending mode.
If your DeamonSet nodeSelector matches 3 nodes, the scheduler would stick to these nodes. There shouldn't be Pods stuck in Pending.
Both solutions sounds good. Maybe using a Deployment + affinities would be more reliable: with a DaemonSet, if one of your node fails, the third replica would not be re-scheduled until you fix that node. Which may or may not be an issue ... up to you.