I have created a service in GKE exposed using an internal load balancer. The health check port for the ILB is 10256 which is kube-proxy. The port on the ILB is that of my kubernetes service port. (Not the target port). However there is no information as such on the ILB which indicates that the traffic is reaching the NodePort on which the service is exposed. How does this work internally ?
The answer is "you don't". When using the Google ILB, traffic isn't forwarded to the NodePort. From Google Compute Engine Docs:
Internal TCP/UDP load balancers are not proxies; they pass traffic to backends on the same port on which the traffic is received
The interesting part is that you don't even need the node port for Google Load Balancers. From Kubernetes Docs
...not strictly required on all cloud providers (e.g. Google Compute Engine does not need to allocate a NodePort to make LoadBalancer work, but AWS does) but the current API requires it
So to answer your question on how it works, it's done in IP Tables by matching the external IP Address of the Service. You can follow the chain from service ip address to pod ip address and port using iptables-save (Google Container Optimized OS doesn't have iptables-save, ubuntu does)
$ sudo iptables-save | grep 35.221.57.238
-A KUBE-SERVICES -d 35.221.57.238/32 -p tcp -m comment --comment "kuard/kuard: loadbalancer IP" -m tcp --dport 8081 -j KUBE-FW-VUXQQGPA6IZX7OIE
$ sudo iptables-save | grep KUBE-FW-VUXQQGPA6IZX7OIE
:KUBE-FW-VUXQQGPA6IZX7OIE - [0:0]
-A KUBE-FW-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard: loadbalancer IP" -j KUBE-MARK-MASQ
-A KUBE-FW-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard: loadbalancer IP" -j KUBE-SVC-VUXQQGPA6IZX7OIE
-A KUBE-FW-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard: loadbalancer IP" -j KUBE-MARK-DROP
-A KUBE-SERVICES -d 35.221.57.238/32 -p tcp -m comment --comment "kuard/kuard: loadbalancer IP" -m tcp --dport 8081
$ sudo iptables-save | grep KUBE-SVC-VUXQQGPA6IZX7OIE
:KUBE-SVC-VUXQQGPA6IZX7OIE - [0:0]
-A KUBE-FW-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard: loadbalancer IP" -j KUBE-SVC-VUXQQGPA6IZX7OIE
-A KUBE-NODEPORTS -p tcp -m comment --comment "kuard/kuard:" -m tcp --dport 31297 -j KUBE-SVC-VUXQQGPA6IZX7OIE
-A KUBE-SERVICES -d 10.31.251.41/32 -p tcp -m comment --comment "kuard/kuard: cluster IP" -m tcp --dport 8081 -j KUBE-SVC-VUXQQGPA6IZX7OIE
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.10000000009 -j KUBE-SEP-7UUYG24J2OBFZYRF
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.11110999994 -j KUBE-SEP-6ANDSGXDBEGTAG23
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.12500000000 -j KUBE-SEP-25IXXMCCF3XEBS6O
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.14286000002 -j KUBE-SEP-NHB6VFRNINLBDDVN
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.16667000018 -j KUBE-SEP-CORNB7YN4D5QTBEL
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.20000000019 -j KUBE-SEP-TVLRYFRKJELCUDJW
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.25000000000 -j KUBE-SEP-GZREDYSZ5AHV77PW
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-QJSD2YXPKN3UWYMO
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-GOWSUJ5VHTYBPFCG
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -j KUBE-SEP-HFAJBHAOVDISHRVT
$ sudo iptables-save | grep KUBE-SEP-7UUYG24J2OBFZYRF
:KUBE-SEP-7UUYG24J2OBFZYRF - [0:0]
-A KUBE-SEP-7UUYG24J2OBFZYRF -s 10.28.25.11/32 -m comment --comment "kuard/kuard:" -j KUBE-MARK-MASQ
-A KUBE-SEP-7UUYG24J2OBFZYRF -p tcp -m comment --comment "kuard/kuard:" -m tcp -j DNAT --to-destination 10.28.25.11 :8080
-A KUBE-SVC-VUXQQGPA6IZX7OIE -m comment --comment "kuard/kuard:" -m statistic --mode random --probability 0.10000000009 -j KUBE-SEP-7UUYG24J2OBFZYRF