K
Q

Question

unable to call services from other nodes

2/10/2017

I have a kubernetes cluster with one master and 2 nodes. The Dashboard is running on node 1 with docker ip 10.244.15.2:9090. I can curl the dashboard from node 1 but neither from master, api or node 2.

$ kubectl --namespace kube-system get svc
NAME                   CLUSTER-IP     EXTERNAL-IP   PORT(S)         AGE
kube-dns               10.100.0.10    <none>        53/UDP,53/TCP   2m
kubernetes-dashboard   10.100.70.70   <none>        80/TCP          2m

http://localhost:8001/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard in browser with proxy on localhost ->

Error: 'dial tcp 10.244.15.2:9090: getsockopt: connection timed out'
Trying to reach: 'http://10.244.15.2:9090/'

when i traceroute the dashboard from master the packages drop at node 1.

traceroute to 10.244.15.2 (10.244.15.2), 30 hops max, 60 byte packets
 1  172.17.8.64 (172.17.8.64)  0.227 ms  0.127 ms  0.171 ms
 2  * * *

curl from node 1 (traceroute 10.100.70.70 ends in wan, how does curl end at the service?!)

core@node-01 ~ $ curl 10.100.70.70                                                                                                                            

     <!doctype html> <html ng-app="kubernetesDashboard"> <head> <meta charset="utf-8"> <title>Kubernetes Dashboard</title> <link rel="icon" type="image/png" href="assets/images/kubernetes-logo.png"> <meta name="viewport" content="width=device-width"> <link rel="stylesheet" href="static/vendor.36bb79bb.css"> <link rel="stylesheet" href="static/app.b9ddff98.css"> </head> <body> <!--[if lt IE 10]>
          <p class="browsehappy">You are using an <strong>outdated</strong> browser.
          Please <a href="http://browsehappy.com/">upgrade your browser</a> to improve your
          experience.</p>
        <![endif]--> <kd-chrome layout="column" layout-fill> </kd-chrome> <script src="static/vendor.633c6c7a.js"></script> <script src="api/appConfig.json"></script> <script src="static/app.64903baa.js"></script> </body> </html>

iptables on node 1

core@node-01 ~ $ sudo iptables -L -n  -t nat                                                                                                                  
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
RETURN     all  --  10.244.0.0/16        10.244.0.0/16       
MASQUERADE  all  --  10.244.0.0/16       !224.0.0.0/4         
MASQUERADE  all  -- !10.244.0.0/16        10.244.0.0/16       

Chain KUBE-MARK-DROP (0 references)
target     prot opt source               destination         
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (4 references)
target     prot opt source               destination         
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-3FFGH6DHFBTFHQWP (2 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  172.17.8.101         0.0.0.0/0            /* default/kubernetes:https */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: SET name: KUBE-SEP-3FFGH6DHFBTFHQWP side: source mask: 255.255.255.255 tcp to:172.17.8.101:443

Chain KUBE-SEP-BOVPSCUJOBAVHYQ3 (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.244.65.3          0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */ tcp to:10.244.65.3:53

Chain KUBE-SEP-DXV3B2UH7M4BGYEA (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.244.65.3          0.0.0.0/0            /* kube-system/kube-dns:dns */
DNAT       udp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */ udp to:10.244.65.3:53

Chain KUBE-SEP-MNI6KNBAY3B2CO64 (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.244.65.2          0.0.0.0/0            /* kube-system/kubernetes-dashboard: */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kubernetes-dashboard: */ tcp to:10.244.65.2:9090

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.100.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-TCOU7JCQXEZGVUNU  udp  --  0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
KUBE-SVC-ERIFXISQEP7F7OF4  tcp  --  0.0.0.0/0            10.100.0.10          /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
KUBE-SVC-XGLOHA7QRQ3V22RZ  tcp  --  0.0.0.0/0            10.100.70.70         /* kube-system/kubernetes-dashboard: cluster IP */ tcp dpt:80
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
target     prot opt source               destination         
KUBE-SEP-BOVPSCUJOBAVHYQ3  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns-tcp */

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target     prot opt source               destination         
KUBE-SEP-3FFGH6DHFBTFHQWP  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-3FFGH6DHFBTFHQWP side: source mask: 255.255.255.255
KUBE-SEP-3FFGH6DHFBTFHQWP  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
target     prot opt source               destination         
KUBE-SEP-DXV3B2UH7M4BGYEA  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kube-dns:dns */

Chain KUBE-SVC-XGLOHA7QRQ3V22RZ (1 references)
target     prot opt source               destination         
KUBE-SEP-MNI6KNBAY3B2CO64  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-system/kubernetes-dashboard: */

ip route on node 1

core@node-01 ~ $ ip route                                                                                                                                                   
default via 172.17.8.1 dev eth1  proto dhcp  src 172.17.8.64  metric 1024 
default via 192.168.121.1 dev eth0  proto dhcp  src 192.168.121.17  metric 1024 
10.244.15.0/24 dev docker0  proto kernel  scope link  src 10.244.15.1 
10.244.98.0/24 via 172.17.8.101 dev eth1 
10.244.100.0/24 via 172.17.8.103 dev eth1 
172.17.8.0/24 dev eth1  proto kernel  scope link  src 172.17.8.102 
172.17.8.1 dev eth1  proto dhcp  scope link  src 172.17.8.64  metric 1024 
192.168.121.0/24 dev eth0  proto kernel  scope link  src 192.168.121.17 
192.168.121.1 dev eth0  proto dhcp  scope link  src 192.168.121.17  metric 1024

what is wrong here or how could i proceed debugging?

-- Milleu

kubernetes

networking

K
Q

unable to call services from other nodes

Similar Questions

0 Answers

KQ

unable to call services from other nodes

Similar Questions

0 Answers

K
Q