My gateway service does dns resolution once and then caches the IP address. For example, if I have a kubernetes Service called service-a, and the gateway routes to http://service-a/, dns resolution works perfectly. But if later the IP address for service-a changes, my gateway service would not notice, and keep sending traffic to the original IP address.
So: will the internal cluster IP of services stay constant, even if I change the service by changing its selectors, or adding an external load balancer, etc?
Yes, the internal cluster IP of a service will stay the same unless you delete the service or manually change its cluster IP.