I have an app, which I want to run on Kubernetes (currently on AWS ECS). The app has two TCP ports, neither is http. One port, say APORT is common across all the app instances (replicas) and should be load balanced. The other, lets call it a BPORT, however, is specific to this particular instance of the app, e.g. pod/container specific. Now here is my problem: the app register it's BPORT with an external controller and the controller should be able to reach this app via that port. I can use NodePort to expose that NodePort to the external IP. From my pod, I will obtain a value of that NodePort and register with the external controller. However, a service only assigns single NodePort across all replicas, so if I want multiple replicas, I have to run multiple services. Running multiple services presents a problem on the APORT side, as this port should be load balanced, ideally sitting behind IPVS, and as far as I understand IPVS does not allow to LB between multiple services. Another wrinkle that ideally I would like to be able to add more replicas to scale this whole thing without service interruption/restarts. Any ideas? Thanks!
Kubernetes doesn't really handle this, or at least it doesn't get involved. You can use a Service object for A port like normal, but for the B port, you wouldn't use a Service at all, things would have to directly use the pod IP instead, just like if these were servers rather than containers.