We have had our database running on Kubernetes cluster (deployed to our private network) in Google cloud for a few months now. Last week we noticed that for some reason the IP address of all underlying nodes (VMs) changed. This caused an outage. We have been using the NodePort configuration of Kubernetes for our service to access our database (https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). We understand that the IP address of the pods within the VMs are dynamic and will eventually change, however we did not know that the IP address of the actual nodes (VMs) may also change. Is this normal? Does anyone know what can cause a VM IP address change in a Kubernetes cluster?
If you are using a managed Kubernetes Engine (GKE) cluster, this is expected as nodes are mortal and might be replaced or restarted if it becomes unresponsive for example. Therefore the node's IP will change. There is currently no way to assign a static (fixed) public IP to nodes. In this case you should expose your DB service as cluster IP instead. it will have an unchanged static IP. Here's an example on how to do that.
Alternatively, if you are using a non-managed kubernetes cluster in Compute Engine (GCE) then you simply have to promote your nodes IP's to static.
From the documentation about Ephemeral IP Addresses on GCP,
When you create an instance or forwarding rule without specifying an IP address, the resource is automatically assigned an ephemeral external IP address. Ephemeral external IP address are released from a resource if you delete the resource. For VM instances, if you stop the instance, the IP address is also released. Once you restart the instance, it is assigned a new ephemeral external IP address.
You can assign static external IP addresses to instances, but as @Notauser mentioned, it is not recommended for Kubernetes nodes. This is because you may configure autoscaler for your instance groups and node sizes can be minimized or maximized. Also, you need to reserve a static IP address for each node, which is not recommended. Moreover you will waste Static IP address resources and if the reserved static IP addresses are not used, you will still be charged for that.
Otherwise you can configure HTTP loadbalancer using ingress and then reserve a static IP address for your load balancer. Instead of using NodePort you should use ClusterIP type services and create an ingress rule forwarding the traffic to those services.