We are running kubernetes 1.5.7 on CoreOS in AWS. Our kube-dns image versions are

gcr.io/google_containers/kubedns-amd64:1.9 gcr.io/google_containers/kube-dnsmasq-amd64:1.4.1

The args that we pass to dnsmasq are

  --cache-size=1000
  --no-resolv
  --server=/in-addr.arpa/ip6.arpa/cluster.local/ec2.internal/127.0.0.1#10053
  --server=169.254.169.253
  --server=8.8.8.8
  --log-facility=-
  --log-async
  --address=/com.cluster.local/com.svc.cluster.local/com.kube-system.svc.cluster.local/<ourdomain>.com.cluster.local/<ourdomain>.com.svc.cluster.local/<ourdomain>.com.kube-system.svc.cluster.local/com.ec2.internal/ec2.internal.kube-system.svc.cluster.local/ec2.internal.svc.cluster.local/ec2.internal.cluster.local/

We run 1 kube-dns pod per node in 20 node clusters. For the last few months we have been experiencing DNS failures that range from a 5 - 10 minute event that renders our services mostly unusable because name resolution is failing for most name lookups. During these events we were running 3 - 6 kube-dns pods. Since then we have drastically over provisioned our kube-dns pods to 1 per node and have not seen any of the long 5 - 10 minute DNS failure events. However now we are still seeing smaller DNS failure events that range from 1 - 30 seconds. During the investigation of these issues we noticed in our logs the following errors from the dnsmasq-metrics container

ERROR: logging before flag.Parse: W0517 03:19:50.139060 1 server.go:53] Error getting metrics from dnsmasq: read udp 127.0.0.1:36181->127.0.0.1:53: i/o timeout

When ever we have one of our smaller DNS events lasting 1 - 30 seconds we find the these logs from the kube-dns pods. For awhile we were suspecting that we were experiencing an iptables/conntrack problem wrt to pods hitting the kube-dns service. But based off these dnsmasq related errors we believe dnsmasq is refusing connections for some period of time causing the DNS failures we have been experiencing. For people who are not familiar with the dnsmasq-metrics container it is performing DNS lookups against the dnsmasq container in the same pod to get dnsmasq stats. If the dnsmasq stats cannot be retrieved via a DNS lookup it seems logical to think that services performing a DNS lookup could experience the same problem.

It's worth noting that during these issues we do NOT see the following logs from dnsmasq which makes me believe we are not hitting this threshold.

dnsmasq: Maximum number of concurrent DNS queries reached (max: 150)

I feel pretty confident that our current DNS errors are related to dnsmasq refusing connections intermittently. I'm curious if other users are seeing the same problems where the kube-dns pod logs the error from dnsmasq-metrics and during that same time frame DNS errors are logged from applications in the cluster.

Additionally if anyone has any ideas on what to do next to find out exactly what is happening wrt dnsmasq refusing connections. I'm pondering if it would be useful to run dnsmasq in debug mode but also worried that will introduce other problems related to running in debug mode. Other options we are considering is slowly rolling out CoreDNS (https://github.com/coredns/coredns).