How to configure kube-proxy to serve from local node only

11/2/2018

I need to configure kube-proxy to server from the pods running on the current node only , and avoid for the connections to bounce around different nodes.

-- Ijaz Ahmad Khan
kube-proxy
kubernetes

2 Answers

11/5/2018

Found a solution in docs:

As of Kubernetes 1.5, packets sent to Services with Type=NodePort are source NAT’d by default. You can test this by creating a NodePort Service:

$ kubectl expose deployment source-ip-app --name=nodeport --port=80 --target-port=8080 --type=NodePort
service/nodeport exposed

$ NODEPORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services nodeport)
$ NODES=$(kubectl get nodes -o jsonpath='{ $.items[*].status.addresses[?(@.type=="ExternalIP")].address }')

If you’re running on a cloudprovider, you may need to open up a firewall-rule for the nodes:nodeport reported above. Now you can try reaching the Service from outside the cluster through the node port allocated above.

$ for node in $NODES; do curl -s $node:$NODEPORT | grep -i client_address; done
client_address=10.180.1.1
client_address=10.240.0.5
client_address=10.240.0.3

Note that these are not the correct client IPs, they’re cluster internal IPs. This is what happens: Client sends packet to node2:nodePort node2 replaces the source IP address (SNAT) in the packet with its own IP address node2 replaces the destination IP on the packet with the pod IP packet is routed to node 1, and then to the endpoint the pod’s reply is routed back to node2 the pod’s reply is sent back to the client

Visually:

          client
             \ ^
              \ \
               v \
   node 1 <--- node 2
    | ^   SNAT
    | |   --->
    v |
 endpoint

To avoid this, Kubernetes has a feature to preserve the client source IP (check here for feature availability). Setting service.spec.externalTrafficPolicy to the value Local will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. If there are no local endpoints, packets sent to the node are dropped, so you can rely on the correct source-ip in any packet processing rules you might apply a packet that make it through to the endpoint. Set the service.spec.externalTrafficPolicy field as follows:

$ kubectl patch svc nodeport -p '{"spec":{"externalTrafficPolicy":"Local"}}'
service/nodeport patched

Now, re-run the test:

$ for node in $NODES; do curl --connect-timeout 1 -s $node:$NODEPORT | grep -i client_address; done
client_address=104.132.1.79

Note that you only got one reply, with the right client IP, from the one node on which the endpoint pod is running. This is what happens: client sends packet to node2:nodePort, which doesn’t have any endpoints packet is dropped client sends packet to node1:nodePort, which does have endpoints node1 routes packet to endpoint with the correct source IP

Visually:

        client
       ^ /   \
      / /     \
     / v       X
   node 1     node 2
    ^ |
    | |
    | v
 endpoint
-- Ijaz Ahmad Khan
Source: StackOverflow

11/5/2018

Referring to the documentation, use the flag:

--bind-address 127.0.0.1

For this flag, you need to add run script of kube-proxy. For systemd, it is stored here:

/etc/systemd/system/kube-proxy.service

than restart the service of kube-proxy:

systemctl restart kube-proxy
-- Nick Rak
Source: StackOverflow