Kubernetes pods can't ping each other using ClusterIP

7/29/2019

I'm trying to ping the kube-dns service from a dnstools pod using the cluster IP assigned to the kube-dns service. The ping request times out. From the same dnstools pod, I tried to curl the kube-dns service using the exposed port, but that timed out as well.

Following is the output of kubectl get pods --all-namespaces -o wide

NAMESPACE       NAME                                      READY   STATUS    RESTARTS   AGE     IP             NODE          NOMINATED NODE
default         pod/busybox                               1/1     Running   62         2d14h   192.168.1.37   kubenode      <none>
default         pod/dnstools                              1/1     Running   0          2d13h   192.168.1.45   kubenode      <none>
default         pod/nginx-deploy-7c45b84548-ckqzb         1/1     Running   0          6d11h   192.168.1.5    kubenode      <none>
default         pod/nginx-deploy-7c45b84548-vl4kh         1/1     Running   0          6d11h   192.168.1.4    kubenode      <none>
dmi             pod/elastic-deploy-5d7c85b8c-btptq        1/1     Running   0          2d14h   192.168.1.39   kubenode      <none>
kube-system     pod/calico-node-68lc7                     2/2     Running   0          6d11h   10.62.194.5    kubenode      <none>
kube-system     pod/calico-node-9c2jz                     2/2     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/coredns-5c98db65d4-5nprd              1/1     Running   0          6d12h   192.168.0.2    kubemaster    <none>
kube-system     pod/coredns-5c98db65d4-5vw95              1/1     Running   0          6d12h   192.168.0.3    kubemaster    <none>
kube-system     pod/etcd-kubemaster                       1/1     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-apiserver-kubemaster             1/1     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-controller-manager-kubemaster    1/1     Running   1          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-proxy-9hcgv                      1/1     Running   0          6d11h   10.62.194.5    kubenode      <none>
kube-system     pod/kube-proxy-bxw9s                      1/1     Running   0          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/kube-scheduler-kubemaster             1/1     Running   1          6d12h   10.62.194.4    kubemaster    <none>
kube-system     pod/tiller-deploy-767d9b9584-5k95j        1/1     Running   0          3d9h    192.168.1.8    kubenode      <none>
nginx-ingress   pod/nginx-ingress-66wts                   1/1     Running   0          5d17h   192.168.1.6    kubenode      <none>

In the above output, why do some pods have an IP assigned in the 192.168.0.0/24 subnet whereas others have an IP that is equal to the IP address of my node/master? (10.62.194.4 is the IP of my master, 10.62.194.5 is the IP of my node)

This is the config.yml I used to initialize the cluster using kubeadm init --config=config.yml

apiServer:
certSANs:
- 10.62.194.4
extraArgs:
    authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: dev-cluster
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
    dataDir: /var/lib/etcd
imageRepository: k8s.gcr.io
kind: ClusterConfiguration
kubernetesVersion: v1.15.1
networking:
dnsDomain: cluster.local
podSubnet: 192.168.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}

Result of kubectl get svc --all-namespaces -o wide

NAMESPACE     NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE     SELECTOR
default       service/kubernetes      ClusterIP   10.96.0.1       <none>        443/TCP                  6d12h   <none>
default       service/nginx-deploy    ClusterIP   10.97.5.194     <none>        80/TCP                   5d17h   run=nginx
dmi           service/elasticsearch   ClusterIP   10.107.84.159   <none>        9200/TCP,9300/TCP        2d14h   app=dmi,component=elasticse
dmi           service/metric-server   ClusterIP   10.106.117.2    <none>        8098/TCP                 2d14h   app=dmi,component=metric-se
kube-system   service/calico-typha    ClusterIP   10.97.201.232   <none>        5473/TCP                 6d12h   k8s-app=calico-typha
kube-system   service/kube-dns        ClusterIP   10.96.0.10      <none>        53/UDP,53/TCP,9153/TCP   6d12h   k8s-app=kube-dns
kube-system   service/tiller-deploy   ClusterIP   10.98.133.94    <none>        44134/TCP                3d9h    app=helm,name=tiller

The command I ran was kubectl exec -ti dnstools -- curl 10.96.0.10:53

EDIT:

I raised this question because I got this error when trying to resolve service names from within the cluster. I was under the impression that I got this error because I cannot ping the DNS server from a pod.

Output of kubectl exec -ti dnstools -- nslookup kubernetes.default

;; connection timed out; no servers could be reached

command terminated with exit code 1

Output of kubectl exec dnstools cat /etc/resolv.conf

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local reddog.microsoft.com
options ndots:5

Result of kubectl get ep kube-dns --namespace=kube-system

NAME       ENDPOINTS                                                  AGE
kube-dns   192.168.0.2:53,192.168.0.3:53,192.168.0.2:53 + 3 more...   6d13h

EDIT: Ping-ing the CoreDNS pod directly using its Pod IP times out as well:

/ # ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes
^C
--- 192.168.0.2 ping statistics ---
24 packets transmitted, 0 packets received, 100% packet loss

EDIT: I think something has gone wrong when I was setting up the cluster. Below are the steps I took when setting up the cluster:

  1. Edit host files on master and worker to include the IP's and hostnames of the nodes
  2. Disabled swap using swapoff -a and disabled swap permanantly by editing /etc/fstab
  3. Install docker prerequisites using apt-get install apt-transport-https ca-certificates curl software-properties-common -y
  4. Added Docker GPG key using curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
  5. Added Docker repo using add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
  6. Install Docker using apt-get update -y; -get install docker-ce -y
  7. Install Kubernetes prerequisites using curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
  8. Added Kubernetes repo using echo 'deb http://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list
  9. Update repo and install Kubernetes components using apt-get update -y; apt-get install kubelet kubeadm kubectl -y

Configure master node:

  1. kubeadm init --apiserver-advertise-address=10.62.194.4 --apiserver-cert-extra-sans=10.62.194.4 --pod-network-cidr=192.168.0.0/16
  2. Copy Kube config to $HOME: mkdir -p $HOME/.kube; sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config; sudo chown $(id -u):$(id -g) $HOME/.kube/config
  3. Installed Calico using kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml; kubectl apply -f https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

On node: On the node I did the kubeadm join command using the command printed out from kubeadm token create --print-join-command on the master

-- user3583252
kubernetes

4 Answers

7/29/2019

note that control plane components like api server, etcd that runs on master node are bound to host network. and hence you see the ip address of the master server.

On the other hand, the apps that you deployed are going to get the ips from the pod subnet range. those vary from cluster node ip's

Try below steps to test dns working or not

deploy nginx.yaml

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: web
  labels:
    app: nginx
spec:
  serviceName: "nginx"
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: k8s.gcr.io/nginx-slim:0.8
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
      volumes:
      - name: www
        emptyDir:
kuebctl create -f nginx.yaml

master $ kubectl get po
NAME      READY     STATUS    RESTARTS   AGE
web-0     1/1       Running   0          1m
web-1     1/1       Running   0          1m

master $ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   35m
nginx        ClusterIP   None         <none>        80/TCP    2m

master $ kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
Address 2: 10.40.0.2 web-1.nginx.default.svc.cluster.local
/ #


/ # nslookup web-0.nginx
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local


/ # nslookup web-0.nginx.default.svc.cluster.local
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      web-0.nginx.default.svc.cluster.local
Address 1: 10.40.0.1 web-0.nginx.default.svc.cluster.local
-- P Ekambaram
Source: StackOverflow

7/29/2019

On some cases the local host that Elasticsearch publishes is not routable/accessible from other hosts. On these cases you will have to configure network.publish_host in the yml config file, in order for Elasticsearch to use and publish the right address.

Try configuring network.publish_host to the right public address.

See more here: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings

-- Opster Elasticsearch Support
Source: StackOverflow

7/29/2019

The kubernetes system pods get assigned the host ip since they provide low level services that are not dependant on an overlay network (or in case of calico even provide the overlay network). They have the ip of the node where they run.

A common pod uses the overlay network and gets assigned an ip from the calico range, not from the metal node they run on.

You can't access DNS (port 53) with HTTP using curl. You can use dig to query a DNS resolver. A service IP is not reachable by ping since it is a virtual IP just used as a routing handle for the iptables rules setup by kube-proxy, therefore a TCP connection works, but ICMP not. You can ping a pod IP though, since it is assigned from the overlay network.

-- Thomas
Source: StackOverflow

7/29/2019

You should check on the same namespace

Currently, you are in default namespace and curl to other kube-system namespace.

You should check in the same namespace, I think it works.

enter image description here

-- Thanh Nguyen Van
Source: StackOverflow