I have a cluster which spans over multiple datacenters. I am searching for a way to deploy an application evenly over these datacenters, so that i have some kind of high-availability when one of the datacenters die.

I understand that i can label my nodes with failure-domain.beta.kubernetes.io/zone=, according to the datacenters, and the scheduler will spread the pods evenly across the zones with the help of the SelectorSpreadPriority.

However, i am unsure about recovery after a failure. If one of the datacenters die, the Replica Manager will spin up another few pods to compensate for the unreachable pods which were placed in the crashed datacenter. Of course, these new pods are placed in the remaining datacenters, since the Scheduler works with best-effort, and currently only sees one failure-domain, since the other one is crashed.

As a result, all pods in this Replica Set are running in the same datacenters. When the crashed datacenter comes online again, how can i redistribute the running pods, so that some of the pods from the surviving datacenter are moved back to the revived datacenter to get high-availability back again? Or is this done automatically?