Today I encount a problem,Apache Flink(1.10) taskmanager's pod(may be we could think it it any pod run k8s) in node uat-001 cloud not access jobmanager's pod 6123 port in azshara-k8s03. I login pod in azshara-k8s02 and scan the jobmanager's pod like this:

$ ./pod-logging.sh
+ kubectl get pods -n dabai-fat
+ grep soa-wallet-service
soa-wallet-service-5666cbd565-pgcph                     1/1     Running                    0          5d19h
++ kubectl get pod -l k8s-app=soa-wallet-service -n dabai-fat -o 'jsonpath={.items[0].metadata.name}'
+ POD=soa-wallet-service-5666cbd565-pgcph
+ kubectl exec -it soa-wallet-service-5666cbd565-pgcph /bin/ash -n dabai-fat
/ # nmap -T4 10.254.58.171
Starting Nmap 7.70 ( https://nmap.org ) at 2020-05-02 04:41 UTC
Nmap scan report for flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171)
Host is up (0.000018s latency).
Not shown: 992 closed ports
PORT      STATE SERVICE
22/tcp    open  ssh
80/tcp    open  http
111/tcp   open  rpcbind
443/tcp   open  https
6123/tcp  open  backup-express
8081/tcp  open  blackice-icecap
8443/tcp  open  https-alt
30000/tcp open  ndmps

Nmap done: 1 IP address (1 host up) scanned in 1.57 seconds

the port 6123 is opening, and I login the pod taskmanager in uat-001, the scan result like this:

root@flink-taskmanager-54d85f57c7-wd2nb:/opt/flink# nmap -T4 10.254.58.171
Starting Nmap 7.70 ( https://nmap.org ) at 2020-05-02 04:40 UTC
Nmap scan report for flink-jobmanager.dabai-fat.svc.cluster.local (10.254.58.171)
Host is up (0.00024s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
111/tcp  open  rpcbind
3306/tcp open  mysql
8443/tcp open  https-alt

Nmap done: 1 IP address (1 host up) scanned in 1.51 seconds

no 6123 port, but the network is fine. How did this happen? What situation could come up this problem?