microk8s clusterIP service appear to be doing source NAT when the documentation says it should not

2/10/2020

I am having a networking issue in Kubernetes. I am trying to preserve the source IP of incoming requests to a clusterIP service, but I find that the requests appear to be source NAT'd. That is, they carry the IP address of the node as the source IP rather than the IP of the pod making the request. I am following the example for cluster IPs here: https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-clusterip but I find that the behavior of Kubernetes is totally different for me. The above example has me deploy an echo server which reports the source IP. This is deployed behind a clusterIP service which I request from a separate pod running busybox. The response from the echo server is below:

CLIENT VALUES:
client_address=10.1.36.1
command=GET
real path=/
query=nil
request_version=1.1
request_uri=http://10.152.183.99:8080/
SERVER VALUES:
server_version=nginx: 1.10.0 - lua: 10001
HEADERS RECEIVED:
connection=close
host=10.152.183.99
user-agent=Wget
BODY

The source IP 10.1.36.1 belongs to the node. I expected to see the address of busybox which is 10.1.36.168. Does anyone know why SNAT would be enabled for a clusterIP? It's really strange to me that this directly contradicts the official documentation. (edited)

All of this is running on the same node. The node is running in iptables mode. I am using microk8s.

My microk8s version:

Client:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
Server:
  Version:  v1.2.5
  Revision: bb71b10fd8f58240ca47fbb579b9d1028eea7c84

Output of kubectl describe service clusterip:

Name:              clusterip
Namespace:         default
Labels:            app=source-ip-app
Annotations:       <none>
Selector:          app=source-ip-app
Type:              ClusterIP
IP:                10.152.183.106
Port:              <unset>  80/TCP
TargetPort:        8080/TCP
Endpoints:         10.1.36.225:8080
Session Affinity:  None
Events:            <none>

Output of kubectl describe pod source-ip-app-7c79c78698-xgd5w:

Name:         source-ip-app-7c79c78698-xgd5w
Namespace:    default
Priority:     0
Node:         riley-virtualbox/10.0.2.15
Start Time:   Wed, 12 Feb 2020 09:19:18 -0600
Labels:       app=source-ip-app
              pod-template-hash=7c79c78698
Annotations:  <none>
Status:       Running
IP:           10.1.36.225
IPs:
  IP:           10.1.36.225
Controlled By:  ReplicaSet/source-ip-app-7c79c78698
Containers:
  echoserver:
    Container ID:   containerd://6775c010145d3951d067e3bb062bea9b70d305f96f84aa870963a8b385a4a118
    Image:          k8s.gcr.io/echoserver:1.4
    Image ID:       sha256:523cad1a4df732d41406c9de49f932cd60d56ffd50619158a2977fd1066028f9
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 12 Feb 2020 09:19:23 -0600
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-7pszf (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-7pszf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-7pszf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From                       Message
  ----    ------     ----       ----                       -------
  Normal  Scheduled  <unknown>  default-scheduler          Successfully assigned default/source-ip-app-7c79c78698-xgd5w to riley-virtualbox
  Normal  Pulled     2m58s      kubelet, riley-virtualbox  Container image "k8s.gcr.io/echoserver:1.4" already present on machine
  Normal  Created    2m55s      kubelet, riley-virtualbox  Created container echoserver
  Normal  Started    2m54s      kubelet, riley-virtualbox  Started container echoserver
-- Riley Wood
kubernetes
microk8s

0 Answers