Configuring a Nginx in front of my front and back end on Kubernetes

7/18/2017

I have been having problems trying to deploy my web app in kubernetes.

I wanted to mimic old deploy with nginx working as reverse proxy in front of my back and front end services.

I have 3 pieces in my system, nginx, front and back. I built 3 deploys, 3 services and exposed only my nginx service using nodePort: 30050.

Without further delays, this is my nginx.conf:

upstream my-server {
    server myserver:3000;
}

upstream my-front {
    server myfront:4200;
}

server {
    listen 80;

    server_name my-server.com;

    location /api/v1 {
        proxy_pass http://my-server;
    }

    location / {
        proxy_pass http://my-front;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;
        proxy_set_header X-Forwarded-For $remote_addr;
    }        
} 

I tried to install curl and nslookup inside one of the pods and tried to do manual request on cluster internal endpoints... tears came to my eyes, everything is working...i am almost a developer worthy of the cloud.

Everything is working smoothly...everything but nginx DNS resolution.

If i do kubectl exec -it my-nginx-pod -- /bin/bash and try to curl 1 of the other 2 services: curl myfront:4200 it works properly.

If i try to nslookup one of them it works as well.

After this i tried to replace, in nginx.conf, the service names with the pods IPs. After restarting the nginx service everything was working.

Why doesn't nginx resolve the upstream names properly? I am going nuts over this.

-- Tiago Bértolo
dns
kubernetes
nginx

1 Answer

7/18/2017

Nginx caches the resolved IPs. To force Nginx to resolve DNS, you can introduce a variable:

location /api/v1 {
    set $url "http://my-server";
    proxy_pass $url;
}

More details can be found in this related this answer.

As it is likely a caching in Nginx, what you describe, it would also explain why restarting (or reloading) Nginx will fix the problem. At least for a while until the DNS entry changes, again.

I think, it is not related to Kubernetes. I had the same problem a while ago when Nginx cached DNS entries of AWS ELBs, which frequently change IPs.

-- Philipp Claßen
Source: StackOverflow