I cannot resolve service from kubernetes.

kubectl get pods -l k8s-app=kube-dns --namespace kube-system
NAME                       READY   STATUS    RESTARTS   AGE   IP
coredns-86c58d9df4-gn62b   1/1     Running   0          18d   10.244.0.58
coredns-86c58d9df4-svmk5   1/1     Running   0          18d   10.244.0.59

containers do not resolve any domains, including kubernetes.default

kubectl exec -ti busybox -- sh
/ # nslookup kubernetes.
defaultServer:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'kubernetes.default'
command terminated with exit code 1

Logs from dns pods do not show any queries (note coredns is configured to log queries)

kubectl logs --namespace=kube-system coredns-86c58d9df4-gn62b
.:53
2019-01-18T21:44:34.271Z [INFO] CoreDNS-1.2.6
2019-01-18T21:44:34.271Z [INFO] linux/amd64, go1.11.2, 756749c
CoreDNS-1.2.6
linux/amd64, go1.11.2, 756749c
 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
[INFO] Reloading
 [INFO] plugin/reload: Running configuration MD5 = 2394cf331ea25e9aacc36ddf69fafcdb
[INFO] Reloading complete
2019-02-04T22:23:21.266Z [INFO] 127.0.0.1:39695 - 58939 "HINFO IN 4718439545634584094.2038959545847864189. udp 57 false 512" NXDOMAIN qr,rd,ra 133 0.021492508s

The kube-node coredns is hosted on is running ubuntu xenial.

I noticed there is a known issue on ubuntu hosts

I applied custom kubelet conf, setting --resolv-conf=/run/systemd/resolve/resolv.conf

❯ systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Wed 2019-02-06 01:05:42 GMT; 5min ago
     Docs: https://kubernetes.io/docs/home/
 Main PID: 27867 (kubelet)
    Tasks: 30 (limit: 4915)
   CGroup: /system.slice/kubelet.service
           └─27867 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-drive --resolv-conf=/run/systemd/resolve/resolv.conf

However I still cannot resolve any services.