I have a Kubernetes cluster that I setup with kube-aws. I'm trying to run a custom NGINX configuration which uses DNS resolutions to proxy_pass. Here is the NGINX block of code

location /api/v1/lead {
  resolver 10.3.0.10 ipv6=off;
  set $container lead-api;
  proxy_pass http://$container:3000;
}

10.3.0.10 comes from the cluster IP of the DNS service found in Kubernetes. I've also tried 127.0.0.11 which is what we use in the docker-compose/docker environments.

$ kubectl describe --namespace=kube-system service kube-dns
Name:                   kube-dns
Namespace:              kube-system
Labels:                 k8s-app=kube-dns
                        kubernetes.io/cluster-service=true
                        kubernetes.io/name=KubeDNS
Selector:               k8s-app=kube-dns
Type:                   ClusterIP
IP:                     10.3.0.10
Port:                   dns     53/UDP
Endpoints:              10.2.26.61:53
Port:                   dns-tcp 53/TCP
Endpoints:              10.2.26.61:53
Session Affinity:       None

This configuration works well on three different environments which use docker-compose. However I get the following error in the NGINX logs of the Kubernetes cluster

[error] 9#9: *20 lead-api could not be resolved (2: Server failure), client: 10.2.26.0, server: , request: "GET /api/v1/lead/661DF757-722B-41BB-81BD-C7FD398BBC88 HTTP/1.1"

If I run nslookup within the NGINX pod I can resolve the host with the same dns server:

$ kubectl exec nginx-1855584872-kdiwh -- nslookup lead-api
Server:         10.3.0.10
Address:        10.3.0.10#53

Name:   lead-api.default.svc.cluster.local
Address: 10.3.0.167

I don't know if it matters or not, but notice the "server" part of the error is empty. When I look at the pod logs for dnsmasq I don't see anything relevant. If I change the NGINX block to hardcode the proxy_pass then it resolves fine. However, I have other configurations that require dynamic proxy names. I could hard code every upstream this way, but I want to know how to make the DNS resolver work.

location /api/v1/lead {
  proxy_pass http://lead-api:3000;
}