Given a multi-master clustered database running under Kubernetes, what happens when a master node goes down and comes back up again?
The goal is to get this working without any downtime.
Is it possible to configure Kubernetes to retain the same IP address across disconnects (node goes down and comes back up)?
Yes. The general idea is that you have to use StatefulSets to preserve names/IPs, although it's more a standard practice to use names (DNS) instead of IPs.
One example is Cassandra and this is an example on how to deploy a cluster on K8s.
If the node comes back with a different IP address, are multi-master database clusters designed to allow master nodes to change their IP addresses on the fly?
This really depends on your configuration, if you hard-code IP addresses in the config then if there's a change of IP address then the master will not be able to join the cluster. If you use names (DNS) as configuration then it's more likely that the master will rejoin the cluster. Again, this really depends on the specific database that you are using (together with the database capabilities).