I am working on securing my Kubernetes cluster with a TLS connection configured in the ingress rule, which essentially terminates the SSL connection at the load balancer. So far so good.
A question came up about whether it would make sense to secure the connection from the load balancer to each of the services running in Kubernetes cluster. My understanding of how Kubernetes works is that services should be able to go up and come down dynamically with no guarantee that the private IPs remain unchanged, so it does not make sense to try to secure the services with TLS connections. Also, the fact that each of the services cannot be exposed to the public internet directly (my configuration is to configure a single ingress rule and routing rules with Istio will take care the routing to the different services), the security is provided in the networking layer.
Is there anything conceptually wrong with my reasoning? Also, is there other mechanism I should be looking at if I want to improve the security setup of my cluster? Istio Auth is not right for my use case, as I do not have services calling other services at all - all my services do not interact with one another.
By service I presume you refer to the kubernetes Service primitive.
Services are not supposed to go up and down dynamically. What you refer to is the Pod which is ephemeral in nature. To make a Pod "more permanent", a Service is tagged to it. When Pods come and go, kubernetes updates iptables rules to route traffic to the live Pods.
Traffic encryption within the cluster can be achieved by encrypting the traffic between the app and the Ingress (Layer 7), or on the cluster network overlay (Layer 3). See this page for more info.