We are using Kubernetes with Azure as cloud provider. The relevant setup to my question is that we have one loadbalancer and one network security group which is attached to all worker VMs. So basically every time I create a service, it creates a record in LoadBalaner frontend IP configuration, and adds a rule in network security group with specified destination port and Source IP addresses (which restricts from which source IP it can access the VM in which port.)

The problem with this set up is that, if I have a service that uses port 5000 which is open to public IP, and another service that also uses port 5000 but is open to only specific IP, both services are effectively open to public IP, because NSG rules are additive. Note that 5000 port number here does not represent the actual VM node port (although that's what Azure thinks) because it's taken care by kube-proxy in each machine and it will send the traffic to correct VM with corresponding node port. And this is why it makes sense to have two services using same port with different ingress rule set up.

Is there any way I can mitigate this problem? I can't think of any architecture setup I can deal with having different ingress rule for multiple services with same destination port.

Thank you