I'm trying to understand why this particular socat command isn't working in my case where I run it in a IPv6 only Kubernetes cluster.
Cluster is build on top of AWS with Calico CNI & containerd. Provisioned using kubeadm and Kubernetes 1.21.
I have run the following socat command which binds to loopback interface ::1,
kubectl --context=$CLUSTER1 run --image=alpine/socat socat -- tcp6-listen:15000,bind=\[::1\],fork,reuseaddr /dev/nullAnd then I try to port-forward and curl to 15000 port,
kubectl --context=$CLUSTER1 port-forward pod/socat 35000:15000 --address=::1
curl -ivg http://localhost:35000I get the error,
Forwarding from [::1]:35000 -> 15000
Handling connection for 35000
E0830 17:09:59.604799 79802 portforward.go:400] an error occurred forwarding 35000 -> 15000: error forwarding port 15000 to pod a8ba619774234e73f4c1b4fe4ff47193af835cffc56cb6ad1a8f91e745ac74e9, uid : failed to execute portforward in network namespace "/var/run/netns/cni-8bade2c1-28c9-6776-5326-f10d55fd0ff9": failed to dial 15000: dial tcp4 127.0.0.1:15000: connect: connection refusedIts listening to 15000 as,
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 ::1:15000 :::* LISTEN 1/socat
However if I run the following it works fine,
kubectl --context=$CLUSTER1 run --image=alpine/socat socat -- tcp6-listen:15000,bind=\[::\],fork,reuseaddr /dev/nullNot sure I understand why port-forward would fail for the loopback interface binding ::1 but not for catch all ::. Can someone please shed some light on this ?
For those of you running into a similar issue with your IPv6 only Kubernetes clusters heres what I have investigated found so far.
Background: It seems that this is a generic issue relating to IPv6 and CRI.
I was running containerd in my setup and containerd versions 1.5.0-1.5.2 added two PRs (don't use socat for port forwarding and use happy-eyeballs for port-forwarding) which fixed a number of issues in IPv6 port-forwarding.
Potential fix: Further to pulling in containerd version 1.5.2 (as part of Ubuntu 20.04 LTS) I was also getting the error IPv4: dial tcp4 127.0.0.1:15021: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found when port-forwarding. This is caused by a DNS issue when resolving localhost. Hence I added localhost to resolve as ::1 in the host machine with the following command.
sed -i 's/::1 ip6-localhost ip6-loopback/::1 localhost ip6-localhost ip6-loopback/' /etc/hostsI think the important point here is that check your container runtimes to make sure IPv6 (tcp6 binding) is supported.