We have two AKS clusters for different environments. Both use a Nginx server as a custom ingress. By that I mean that it acts like an ingress, but it is just a normal Nginx deployment behind a service. There are several good reasons for that setup, the main one being that ingress did not exist in AKS when we started.
The services are defined like this:
apiVersion: v1
kind: Service
metadata:
name: <our name>
namespace: <our namespace>
spec:
ports:
- port: 443
targetPort: 443
selector:
app: <our app>
loadBalancerIP: <our ip>
type: LoadBalancer
externalTrafficPolicy: Local
We have configured Nginx with the real ip module like this:
real_ip_header X-Original-Forwarded-For;
set_real_ip_from 10.0.0.0/8; # or whatever ip is correct
One environment uses the old basic networking, networkPlugin=kubenet. There Nginx logs the real client IP addresses in the log and can use them for access controls. The other uses advanced networking, networkPlugin=azure. There Nginx logs the IP address of one of the nodes, which is useless. Both the X-Original-Forwarded-For and the standard X-Forwarded-For headers are empty and of course the source IP is from the node, not from the client.
Is there a way around this? If at all possible we would like to avoid defining a "real" ingress as our own Nginx server contains custom configuration that would be hard to duplicate in such a setup, plus it is not clear that a standard ingress would help either?
Microsoft should have fixed this by now for real ingresses. However, apparently the fix doesn't cover our case where Nginx runs as a pod behind a service with advanced networking. We were told to use the workaround posted by denniszielke in https://github.com/Azure/AKS/issues/607 where the iptables for all nodes are updated regularly. Quite dirty in my view, but it works.
We still have the service defined as above with "externalTrafficPolicy: Local" and we have installed the ConfigMap and DaemonSet from the link. I changed the script to reduce logging a bit and moved both to another namespace.