Kubernetes UDP Service Connection Problem

12/18/2019

I am trying to resolve a Cluster internal DNS from a Node in the Cluster.
Example: dig @10.96.0.10 kubernetes.local (10.96.0.10 being the Service IP of the DNS)
I am expecting to get the IP of the Service (10.96.0.1 in this case), however the Connection times out.
This Problem only happens if I try connecting from a Host in the Cluster to a Service via UDP, while the Pods of the Service are not hosted on the Node I am connecting from.
If I try to connect from a Pod running on the same Node, it works as expected.
If I try to connect to the Pods directly instead of the Service, it works as expected.
If I try to connect to the Service via TCP instead of UDP, it works as expected.
If I try to connect to the Service when the Pods are running on the same Node as I am connecting from, it works as expected.

I am running Kubernetes v1.17 (Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.0", GitCommit:"70132b0f130acc0bed193d9ba59dd186f0e634cf", GitTreeState:"clean", BuildDate:"2019-12-07T21:12:17Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"}) with the flannel pod network, running on Debian Buster.

So far I have looked at the iptables rules, the service and pod rules seem correct.
The relevant sections of iptables-save:

-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment "kube-system/kube-dns:dns cluster IP" -m udp --dport 53 -j KUBE-SVC-TCOU7JCQXEZGVUNU
-A KUBE-SVC-TCOU7JCQXEZGVUNU -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-SBQ7D3CPOXKXY6NJ
-A KUBE-SVC-TCOU7JCQXEZGVUNU -j KUBE-SEP-CDWMPIYNA34YYC2O
-A KUBE-SEP-CDWMPIYNA34YYC2O -s 10.244.1.218/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-CDWMPIYNA34YYC2O -p udp -m udp -j DNAT --to-destination 10.244.1.218:53
-A KUBE-SEP-SBQ7D3CPOXKXY6NJ -s 10.244.1.217/32 -j KUBE-MARK-MASQ
-A KUBE-SEP-SBQ7D3CPOXKXY6NJ -p udp -m udp -j DNAT --to-destination 10.244.1.217:53

I also used the sudo tcpdump -i flannel.1 udp command on sender and receiver and found out, that the packages get send, but not received.
When I address the Pods directly, for example via dig @10.244.1.218 kubernetes.local, the package gets send and received properly.

-- Lennart Austenfeld
kubernetes
kubernetes-networking

0 Answers