So, I would like to have nginx resolve hostnames for backends at request time. I expect to get HTTP 502 Bad Gateway when back-end service is down and I expect service response, when it's up.

I use nginx:1.15-alpine image for nginx and here is what I have in it's config:

server {

  resolver kube-dns.kube-system.svc.cluster.local valid=5s;

  server_name  mysystem.com;
  listen       80;

  client_max_body_size 20M;

  location = /nginx_status {
      stub_status on;
      access_log off;
  }

  # Services configuration

  location ~ /my-service/ {
      set $service_endpoint http://my-service.namespace:8080;
      proxy_pass $service_endpoint$request_uri;
      include includes/defaults-inc.conf;
      include includes/proxy-inc.conf;
  }

}

So, when I make the request to the nginx, I get 502 Bad Gateway response. Nginx's log say the name is not found:

2018/06/28 19:49:18 [error] 7#7: *1 my-service.namespace could not be resolved (3: Host not found), client: 10.44.0.1, server: mysystem.com, request: "GET /my-service/version HTTP/1.1", host: "35.229.17.63:8080"

However, when I log into the container with shell (kubectl exec ... -- sh) and test the DNS resolution, it works perfectly.

 # nslookup my-service.namespace kube-dns.kube-system.svc.cluster.local
Server:    10.47.240.10
Address 1: 10.47.240.10 kube-dns.kube-system.svc.cluster.local

Name:      my-service.namespace
Address 1: 10.44.0.75 mysystem-namespace-mysystem-namespace-my-service-0.my-service.namespace.svc.cluster.local

Moreover, I can wget http://my-service.namespace:8080/ and get a response.

Why nginx cannot resolve the hostname?

Update: How I managed to resolve it:

In nginx.conf at the server level I have added a resolver setting:

resolver kube-dns.kube-system.svc.cluster.local valid=10s;

Then I used a FQDN in proxy_pass:

proxy_pass http://SERVICE-NAME.YOUR-NAMESPACE.svc.cluster.local:8080;