How do I sniff traffic from the localhost to my k8s cluster (running locally)?

3/12/2021

I have a minikube cluster running locally and a pod, cluster ip 172.17.0.8.

I am using ksniff to sniff traffic on that pod.

In the pod when I ping www.google.com. I can see, in the wireshark capture, ICMP request going to/from:

pod (172.17.0.8) <--> google server (some IP)

I know there is an intermediate step. Where my macbook (the cluster host) is making the request on behalf of the pod and receiving the response to send to the correct pod.

pod (172.17.0.8) <--> cluster host (macbook) <--> google server (some IP)

How can I capture the traffic between the pod and the cluster host (eg. macbook)?

-- mingxiao
kubernetes
network-programming
wireshark

1 Answer

3/14/2021

I haven't used minikube nor a macbook so your mileage may vary, but let's try figuring this out.

From what I've seen, pods are typically provided with a default gateway by the host. In other words, the host serves as a router for the pods it hosts. How do we know that? When running an Ubuntu image on a K8s cluster (and installing iproute2), I get the following routing table:

root@ubuntu:/# ip route sh
default via 10.244.2.1 dev eth0 
10.244.2.0/24 dev eth0 proto kernel scope link src 10.244.2.17

See that default via 10.244.2.1 line? That's the "router" that the host provides. We can even look up its mac address:

root@ubuntu:/# cat /proc/net/arp 
IP address       HW type     Flags       HW address            Mask     Device
10.244.2.5       0x1         0x2         fa:26:3d:e1:10:f5     *        eth0
10.244.2.1       0x1         0x2         8e:f7:54:7b:15:51     *        eth0

So the traffic flow between the pod and the internet is probably like the traffic flow from a host behind a NAT router.

The pod does the arp request/arp response dance, then after learning the "router's" mac address, it'll send IP packets towards the external server with the external server's IP address as the destination and its own IP address as the source. The destination MAC address of these packets would be MAC address of the host's virtual network card. So a capture performed at the pod will actually show you the traffic between the pod and the host as if the host is a router. Once the hosts gets these packets, it probably NATs them in your case and sends them over the internet.

So on the host, if you capture on 10.244.2.1, you should see the same traffic as seen by the pod. But if you capture on the real internet-facing interface, you'll probably see the pod's traffic past the NAT (i.e. with your real IP address as the source IP).

-- Malt
Source: StackOverflow