I have a small Kubernetes on prem cluster (Rancher 2.3.6) consisting of three nodes. The deployments inside the cluster are provisioned dynamically by an external application and always have their replica count set to 1, because these are stateful applications and high availability is not needed.
The applications are exposed to the internet by NodePort services with a random port and ExternalTrafficPolicy set to Cluster. So if the user requests one of the three nodes, the k8s proxy will route and s-NAT the request to the correct node with the application pod.
To this point, everything works fine.
The problem started when we added Applications that rely on the requests source IP. Since the s-NAT replaces the request IP with an internal IP this applications don't work properly.
I know, that setting the services ExternalTrafficPolicy to local will disabke s-natting. But this will also break the architecture, because not every pod has an instance of the application running.
Is there a way to preserve the original client IP and still make use of the internal routing, so i won't have to worry about on which node the request will land?
You can setup MetalLB LoadBalancer in layer2 mode, and use ExternalTrafficPolicy: Local.
In metalLB docs you can read:
When announcing in layer2 mode, one node in your cluster will attract traffic for the service IP. From there, the behavior depends on the selected traffic policy.
In this mode only Nodes which have Service Endpoints (Pod replicas > 0) will serve the incoming traffic; of course in this mode client SourceIPs are preserved.
Web applications can be exposed to outside of cluster using Ingress instead of NodePorts. Ingress object can point to app Deployment using a Service in between, where you can configure service.spec.externalTrafficPolicy: Local to preserve the source IP address. You can have an external load balancer pointing to the nodes of the cluster where ingress controller pods are for traffic routing purposes.
Reference: https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/