I understand that in Kubernetes taint/toleration is used to ensure that a node tainted with a label will only have pods with corresponding toleration to be scheduled on it. Also, through node affinity we (can) ensure that a pod gets scheduled on subset of nodes using labels.
However, I wanted to know if there are use-cases where what can be achieved by one, cannot be achieved by the other. Even scenarios where one might be preferred over the other, will be insightful.
I understand that in Kubernetes taint/toleration is used to ensure that a node tainted with a label will only have pods with corresponding toleration to be scheduled on it.
Yes the node will have pods which tolerate the taint. But the pods which tolerate the taint might also be scheduled to other nodes which doesn't have any taints.
Also, through node affinity we (can) ensure that a pod gets scheduled on subset of nodes using labels.
Yes, with nodeAffinity you can require pods to be scheduled only on the matching nodes.
According to your statement taint/toleration is used to ensure that a node tainted with a label will only have pods with corresponding toleration to be scheduled on it. , but it is not always guaranteed that pod scheduled only that node. For example,
We have 3 nodes, blue, green, red node. and we also have 3 pods B, G, R. Our goal is to place the B pod in the blue node, the R pod in the red node likewise G pod in the green node. We are sharing the same kubernetes cluster with other teams, so there are other pods in the cluster as well as other nodes. We do not want any other pod to be placed on our node. Neither do we want our pods to be placed on their nodes.
If we apply taint on blue node, so that only blue pod can be scheduled on it, similarly apply taint on green node, which tolerate only green pod and red node tolerate red pods.
So, First B pod might be scheduled on blue pod, and G pod scheduled on green pod. and R pod can be ends up on other node.
If we want to solve same problem with node affinity. how can we do that?
We labeled blue node with blue color, red node with red color and green node with green color. We then set node selectors on the pod so that B pod scheduled on blue node, G pod scheduled on green node and R pod scheduled on red node.
So, here problem is, this does not guarantee that other pods are not placed on that node. so there is a chance that other pods may ends up on blue, green or red node.
With combination of taints/toleration and node affinity we can completely dedicated nodes for specific pod.