Kubernetes service cluster ip not accessible but endpoints ip is accessible from within the node

12/26/2018

I setup a single node kubernetes following the kubernetes-the-hard-way guide, except that I'm running on CentOS-7 and I deploy one master and one worker in the same node. I already turn off the firewalld service.

After the installation, I deploy a mongodb service, however the cluster IP is not accessible but the endpoint is accessible. The service detail is as follows:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
kubernetes   ClusterIP   10.254.0.1     <none>        443/TCP     2m
mongodb      ClusterIP   10.254.0.117   <none>        27017/TCP   55s
$ kubectl describe svc mongodb
Name:              mongodb
Namespace:         default
Labels:            io.kompose.service=mongodb
Annotations:       kompose.cmd=kompose convert -f docker-compose.yml
                   kompose.version=1.11.0 (39ad614)
                   kubectl.kubernetes.io/last-applied-configuration= 
{"apiVersion":"v1","kind":"Service","metadata":{"annotations": 
{"kompose.cmd":"kompose convert -f docker-compose.yml","kompose.version":"1.11.0 
(39ad614...
Selector:          io.kompose.service=mongodb
Type:              ClusterIP
IP:                10.254.0.117
Port:              27017  27017/TCP
TargetPort:        27017/TCP
Endpoints:         10.254.0.2:27017
Session Affinity:  None
Events:            <none>

I run mongo 10.254.0.2 on the host, it works, but when I run mongo 10.254.0.117, it does not works. By the way, if I start another mongo pod for example

kubectl run mongo-shell -ti --image=mongo --restart=Never bash

and I tried mongo 10.254.0.2 and mongo 10.254.0.117, they did not work at all.

The kubernetes version I use is 1.10.0.

I think this is a kube-proxy issue, the kube-proxy is configured as follows:

[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://kubernetes.io/docs/concepts/overview/components/#kube- 
proxy https://kubernetes.io/docs/reference/generated/kube-proxy/
After=network.target

[Service]
ExecStart=/usr/local/bin/kube-proxy \
        --config=/var/lib/kubelet/kube-proxy-config.yaml \
        --logtostderr=true \
        --v=2
 Restart=on-failure
 RestartSec=5
 LimitNOFILE=65536

 [Install]
 WantedBy=multi-user.target

and the config file is

kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kubelet/kube-proxy.kubeconfig"
mode: "iptables"
clusterCIDR: "10.254.0.0/16"

This is the ip tables I get

sudo iptables -t nat -nL
    Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
DOCKER     all  --  0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
KUBE-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service portals */
DOCKER     all  --  0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
KUBE-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes postrouting rules */
MASQUERADE  all  --  172.17.0.0/16        0.0.0.0/0           
CNI-0f56c935ec75c77eb189a5fe  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "a54a2f20dbe5d24ec4fb6b059f23aae392cc26853cf2b474a56dff2a2f2d6bb6" */
CNI-d2a650ff06e253010ea31f3d  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "f3252d60a15faa5ff6c4b2aabebdb47aa5652e12c9d874f538b33d6c5913ba47" */
CNI-34b02c799f7bc4e979c15266  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "5a87d86a62dd299e1d36b2ccd631d58896f2724ad9b4e14a93b9dfaa162b09e3" */
CNI-eb80e2736e1009010a27b4b4  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "1891a61e27b764e4a36717166a2b83ce7d2baa5258e54f0ea183c4433b04de38" */
CNI-4d1b80b0072ade1be68c43d1  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "2b90e720350fa78bf6e6756b941526bf181e0b48c6b87207bbc8f097933e67ba" */
CNI-7699fcd0ab82a702bac28bc9  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "3feed2ec479bd17f82cac60adfd1c79c81d4c53d536daa74a46e05f462e2d895" */
CNI-871343dd2a1a9738c94b4dba  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "1a3a7b27889e54494d1e9699efb158dc8f3bb85b147b80db84038c07fd4c9910" */
CNI-3c0d02d02e5aa29b38ada7ba  all  --  10.254.0.0/24        0.0.0.0/0            /* name: "bridge" id: "cdd5d6cf1a772b2acd37471046f53d0aa635733f0d5447a11d76dbb2ee216378" */

Chain CNI-0f56c935ec75c77eb189a5fe (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "a54a2f20dbe5d24ec4fb6b059f23aae392cc26853cf2b474a56dff2a2f2d6bb6" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "a54a2f20dbe5d24ec4fb6b059f23aae392cc26853cf2b474a56dff2a2f2d6bb6" */

Chain CNI-34b02c799f7bc4e979c15266 (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "5a87d86a62dd299e1d36b2ccd631d58896f2724ad9b4e14a93b9dfaa162b09e3" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "5a87d86a62dd299e1d36b2ccd631d58896f2724ad9b4e14a93b9dfaa162b09e3" */

Chain CNI-3c0d02d02e5aa29b38ada7ba (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "cdd5d6cf1a772b2acd37471046f53d0aa635733f0d5447a11d76dbb2ee216378" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "cdd5d6cf1a772b2acd37471046f53d0aa635733f0d5447a11d76dbb2ee216378" */

Chain CNI-4d1b80b0072ade1be68c43d1 (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "2b90e720350fa78bf6e6756b941526bf181e0b48c6b87207bbc8f097933e67ba" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "2b90e720350fa78bf6e6756b941526bf181e0b48c6b87207bbc8f097933e67ba" */

Chain CNI-7699fcd0ab82a702bac28bc9 (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "3feed2ec479bd17f82cac60adfd1c79c81d4c53d536daa74a46e05f462e2d895" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "3feed2ec479bd17f82cac60adfd1c79c81d4c53d536daa74a46e05f462e2d895" */

Chain CNI-871343dd2a1a9738c94b4dba (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "1a3a7b27889e54494d1e9699efb158dc8f3bb85b147b80db84038c07fd4c9910" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "1a3a7b27889e54494d1e9699efb158dc8f3bb85b147b80db84038c07fd4c9910" */

Chain CNI-d2a650ff06e253010ea31f3d (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "f3252d60a15faa5ff6c4b2aabebdb47aa5652e12c9d874f538b33d6c5913ba47" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "f3252d60a15faa5ff6c4b2aabebdb47aa5652e12c9d874f538b33d6c5913ba47" */

Chain CNI-eb80e2736e1009010a27b4b4 (1 references)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            10.254.0.0/24        /* name: "bridge" id: "1891a61e27b764e4a36717166a2b83ce7d2baa5258e54f0ea183c4433b04de38" */
MASQUERADE  all  --  0.0.0.0/0           !224.0.0.0/4          /* name: "bridge" id: "1891a61e27b764e4a36717166a2b83ce7d2baa5258e54f0ea183c4433b04de38" */

Chain DOCKER (2 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           

Chain KUBE-MARK-DROP (0 references)
target     prot opt source               destination         
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x8000

Chain KUBE-MARK-MASQ (4 references)
target     prot opt source               destination         
MARK       all  --  0.0.0.0/0            0.0.0.0/0            MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target     prot opt source               destination         

Chain KUBE-POSTROUTING (1 references)
target     prot opt source               destination         
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-G5V522HWZT6RKRAC (2 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  192.168.56.3         0.0.0.0/0            /* default/kubernetes:https */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: SET name: KUBE-SEP-G5V522HWZT6RKRAC side: source mask: 255.255.255.255 tcp to:192.168.56.3:6443

Chain KUBE-SEP-O34O4OGFBAADOMEG (1 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  all  --  10.254.0.2           0.0.0.0/0            /* default/mongodb:27017 */
DNAT       tcp  --  0.0.0.0/0            0.0.0.0/0            /* default/mongodb:27017 */ tcp to:10.254.0.2:27017

Chain KUBE-SERVICES (2 references)
target     prot opt source               destination         
KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-SVC-NPX46M4PTMTKRN6Y  tcp  --  0.0.0.0/0            10.254.0.1           /* default/kubernetes:https cluster IP */ tcp dpt:443
KUBE-MARK-MASQ  tcp  -- !10.254.0.0/16        10.254.0.117         /* default/mongodb:27017 cluster IP */ tcp dpt:27017
KUBE-SVC-ZDG6MRTNE2LQFT34  tcp  --  0.0.0.0/0            10.254.0.117         /* default/mongodb:27017 cluster IP */ tcp dpt:27017
KUBE-NODEPORTS  all  --  0.0.0.0/0            0.0.0.0/0            /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-NPX46M4PTMTKRN6Y (1 references)
target     prot opt source               destination         
KUBE-SEP-G5V522HWZT6RKRAC  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */ recent: CHECK seconds: 10800 reap name: KUBE-SEP-G5V522HWZT6RKRAC side: source mask: 255.255.255.255
KUBE-SEP-G5V522HWZT6RKRAC  all  --  0.0.0.0/0            0.0.0.0/0            /* default/kubernetes:https */

Chain KUBE-SVC-ZDG6MRTNE2LQFT34 (1 references)
target     prot opt source               destination         
KUBE-SEP-O34O4OGFBAADOMEG  all  --  0.0.0.0/0            0.0.0.0/0            /* default/mongodb:27017 */
-- cgcgbcbc
kube-proxy
kubernetes

1 Answer

12/26/2018

I remove the --network-plugin=cni flag for kubelet service and upgrade the kubernetes to 1.13.0 solve the problem

-- cgcgbcbc
Source: StackOverflow