I've created a kubernetes cluster from scratch as the directions of Kelsey Hightower. As I checked, everything works without any error but after I've deployed my first application, I figured out my application couldn't resolve the DNSes.

I've checked my coredns logs and saw below entries :

.:53
2018/10/16 12:31:45 [INFO] CoreDNS-1.2.2
2018/10/16 12:31:45 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2018/10/16 12:31:45 [INFO] plugin/reload: Running configuration MD5 = 06122de1a2d6c43092ab48d05478dc82
2018/10/16 12:44:27 [ERROR] 2 google.com. A: unreachable backend: read udp 192.168.65.142:51219->172.10.0.2:53: i/o timeout
2018/10/16 12:44:29 [ERROR] 2 google.com. A: unreachable backend: read udp 192.168.65.142:39967->172.10.0.2:53: i/o timeout
2018/10/16 12:44:31 [ERROR] 2 google.com. A: unreachable backend: read udp 192.168.65.142:40187->172.10.0.2:53: i/o timeout

Plus to Kelsey's documentation, I've installed calico on top of Docker.

My Suspect :

I've installed Calico on top of Docker, but my kubelet runtime has been configured to work with Containerd. I'm sshing into calico pod and seems it has network. But none of the containerd pods. But I couldn't find a way to run calico over Containerd.

My Kubelet Service Configuration :

[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=containerd.service
Requires=containerd.service

[Service]
ExecStart=/usr/bin/kubelet \
  --container-runtime=remote \
  --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock \
  --image-pull-progress-deadline=2m \
  --config=/var/lib/kubelet/kubelet-config.yaml \
  --kubeconfig=/var/lib/kubelet/kubeconfig-kubelet \
  --network-plugin=cni \
  --cni-conf-dir=/etc/cni/net.d \
  --cni-bin-dir=/opt/cni/bin \
  --register-node=true \
  --cloud-provider=aws \
  --v=2

Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

And my kubelet config yaml file :

kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: "/etc/kubernetes/pki/ca.pem"
authorization:
  mode: Webhook
clusterDomain: "${cluster_domain}"
clusterDNS:
  - "172.10.0.10"
podCIDR: "172.10.0.0/16"
resolvConf: "/run/systemd/resolve/resolv.conf"
runtimeRequestTimeout: "15m"
tlsCertFile: "/etc/kubernetes/pki/worker.pem"
tlsPrivateKeyFile: "/etc/kubernetes/pki/worker-key.pem"

My resolv.conf file on the node machine :

nameserver 172.10.0.2

I can see that pods could connect to coredns pod but coredns couldn't connect to 172.10.0.2 over port 53. On the host machine, I could telnet to this port and getting answer.

best,