Nginx ingress controller setting unexpected X-Forwarded-* headers

6/10/2021

I'm running into an issue with an nginx ingress controller (ingress-nginx v0.44.0) on EKS where the X-Forwarded-* headers are set to the kubernetes worker node the controller pod is running on as opposed to the details of the request of the actual user hitting the controller itself. As we're terminating our SSL on the ingress controller this means the 'X-Forwarded-Proto' is set to 'http' instead of 'https' which causes issues on the application pods.

I deployed a test pod which returns the headers it received to confirm the issue and I can see these headers having been received:

X-Forwarded-For: <ip of the eks worker node>
X-Forwarded-Host: foo.bar.net
X-Forwarded-Port: 8000
X-Forwarded-Proto: http

I was expecting these though:

X-Forwarded-For: <ip of the origin of the original request>
X-Forwarded-Host: foo.bar.net
X-Forwarded-Port: 443
X-Forwarded-Proto: https

Now, we do have an old legacy cluster running an older nginx ingress controller (nginx-ingress v0.34.1) which does actually behave as I expected, but I'm struggling to find how this has been configured to make it do this correctly. I did notice that the nginx.conf of this controller contains the 'full_x_forwarded_proto' variable identically as described here but I can't find any place where this is configured as in a configmap or similar.

map $http_x_forwarded_proto $full_x_forwarded_proto {
  default $http_x_forwarded_proto;
  "" $scheme;
}

Does anybody have any suggestions how I can configure nginx to send the correct 'X-Forwarded-*' headers?

-- arjen_iw
kubernetes
nginx-ingress

1 Answer

6/10/2021

It depends a lot on the exact networking setup in front of Nginx. By default, Kubernetes routes all external connections through the kube-proxy mesh which hides the true client IP. You also might have an AWS ELB of some kind in front of that which also can hide the client IP depending on settings.

For the first part, see https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-loadbalancer (tl;dr set externalTrafficPolicy: Local) but for the second you'll have to look at your specific load balancer setup.

-- coderanger
Source: StackOverflow