Suppose we have a Kubernetes cluster with two zones, each zone has two nodes(workers, N00 and N01 in Zone 0, N10 and N11 in Zone 1). We want a Kubernetes scheduler that does the following.
(1) When we do a deployment, each zone has one and only one pod. It doesn't matter which node within that zone.
(2) When node N00 with running pod P fails, if N01 within the same zone works, schedule the pod to N01. Otherwise, schedule to another zone, we don't care podAntiAffinity at this time so any node (N10 or N11) in the other zone works.
(3) If N00 was finally restored and P was scheduled on N01, we do nothing at all, but if N00 was restored while P was scheduled on the other zone, we want P to be rescheduled back to its original zone.
Is there any trick that we can play on Kubernetes API to achieve this?
Unfortunately I can't show you a simple way, but I hope the following information will be useful to you.
Using standard scheduler it is doable if you use several similar deployments, one for each zones with different affinity/antiaffinity configuration:
Alternatively, you can write a custom scheduler to add zone selectivity:
Nevertheless, it doesn't allow you to reschedule running pod to another node in different zone when it becomes healthy again.
Kubernetes Scheduler is only responsible for selecting the most suitable node for unassigned pods. It doesn't do anything with pods in the Running state.
There is a design proposal to implement pod rescheduling feature, but it's not ready to use yet.
At the moment, you can take the reschedule project as a starting point to create your own solution.