Unable to get real remote IP in AKS with advanced networking

2/19/2019

We have two AKS clusters for different environments. Both use a Nginx server as a custom ingress. By that I mean that it acts like an ingress, but it is just a normal Nginx deployment behind a service. There are several good reasons for that setup, the main one being that ingress did not exist in AKS when we started.

The services are defined like this:

apiVersion: v1
kind: Service
metadata:
  name: <our name>
  namespace: <our namespace>
spec:
  ports:
  - port: 443
    targetPort: 443
  selector:
    app: <our app>
  loadBalancerIP: <our ip>
  type: LoadBalancer
  externalTrafficPolicy: Local

We have configured Nginx with the real ip module like this:

real_ip_header X-Original-Forwarded-For;
set_real_ip_from 10.0.0.0/8; # or whatever ip is correct

One environment uses the old basic networking, networkPlugin=kubenet. There Nginx logs the real client IP addresses in the log and can use them for access controls. The other uses advanced networking, networkPlugin=azure. There Nginx logs the IP address of one of the nodes, which is useless. Both the X-Original-Forwarded-For and the standard X-Forwarded-For headers are empty and of course the source IP is from the node, not from the client.

Is there a way around this? If at all possible we would like to avoid defining a "real" ingress as our own Nginx server contains custom configuration that would be hard to duplicate in such a setup, plus it is not clear that a standard ingress would help either?

-- ewramner
azure-aks
kubernetes
networking
nginx

1 Answer

3/1/2019

Microsoft should have fixed this by now for real ingresses. However, apparently the fix doesn't cover our case where Nginx runs as a pod behind a service with advanced networking. We were told to use the workaround posted by denniszielke in https://github.com/Azure/AKS/issues/607 where the iptables for all nodes are updated regularly. Quite dirty in my view, but it works.

We still have the service defined as above with "externalTrafficPolicy: Local" and we have installed the ConfigMap and DaemonSet from the link. I changed the script to reduce logging a bit and moved both to another namespace.

-- ewramner
Source: StackOverflow