we are trying to setup a kubernetes cluster manually where controller and masters run in three different Data centre.

Data centreA: ControllerIp: 10.0.1.11 Worker Subnet: 10.0.2.0/24

Data centre B: ControllerIp: 10.1.1.11 Worker Subnet: 10.1.2.0/24

Data centre C: ControllerIp: 10.2.1.11 Worker Subnet: 10.2.2.0/24

now we are able to bring the cluster up and workers are able to join the controller nodes.

root@ip-10-0-1-11:/# kubectl get componentstatus
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}

root@ip-10-0-1-11:/# kubectl get nodes
NAME                            STATUS   ROLES    AGE   VERSION
worker-30838.example.com   Ready    <none>   15h   v1.15.3
worker-9120.example.com    Ready    <none>   15h   v1.15.3

we have created a sample service "hello-world" with 5 replica sets.:

kubectl apply -f https://k8s.io/examples/service/load-balancer-example.yaml
root@ip-10-0-1-11:/# kubectl get deployments
NAME          READY   UP-TO-DATE   AVAILABLE   AGE
hello-world   5/5     5            5           8m30s
nginx         1/1     1            1           15h

post deployment we created the service(NodePort):

root@ip-10-0-1-11:/# kubectl expose deployment hello-world --type=NodePort --name=my-service
root@ip-10-0-1-11:/# kubectl get svc
NAME         TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
kubernetes   ClusterIP   10.32.0.1     <none>        443/TCP          15h
my-service   NodePort    10.32.0.145   <none>        8080:30960/TCP   9m2s

now when we try to connect to the port by using NodePort like below directly to the worker nodes. we intermittently get timed out:

root@ip-10-0-1-11:/# curl http://10.0.2.31:30960




^C
root@ip-10-0-1-11:/# curl http://10.0.2.31:30960
^C
root@ip-10-0-1-11:/# curl http://10.0.2.31:30960
Hello Kubernetes!root@ip-10-0-1-11:/# curl http://10.0.2.31:30960



^C
root@ip-10-0-1-11:/# curl http://10.0.2.31:30960





^C
root@ip-10-0-1-11:/# curl http://10.0.2.31:30960
Hello Kubernetes!root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/# ^C
root@ip-10-0-1-11:/# curl http://10.0.2.31:30960



^C
root@ip-10-0-1-11:/# curl http://10.0.2.31:30960
^C
root@ip-10-0-1-11:/# curl http://10.1.2.41:30960




^C
root@ip-10-0-1-11:/# curl http://10.1.2.41:30960
Hello Kubernetes!root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/# curl http://10.1.2.41:30960
Hello Kubernetes!root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/# curl http://10.1.2.41:30960
Hello Kubernetes!root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/#
root@ip-10-0-1-11:/# curl http://10.1.2.41:30960
curl: (7) Failed to connect to 10.1.2.41 port 30960: Connection timed out

looking forward for suggestion or guidance on this how we can debug or fix the issue.