I have a 3-node cluster (1 master + 2 worker). I am exploring dual-stack capabilities of Kubernetes (version 1.21). I have used kubeadm to initialize this cluster. To begin with, I see IPv4 address of nodes in the this command : command - kubectl get nodes -o wide Also, I can fetch logs of pods using the command: kubectl logs <podname>

Then, I configured all 3 kubelets with --node-ip=IPv6,IPv4. On master, if I run the command - kubectl get nodes -o wide , I can see IPv6 showing up for every node, which is fine. But, on master, if I run the command to fetch logs of a pod, I get the error:

kubectl logs my-nginx-deploy-64fdd68f8d-klg78
Error from server: Get "https://[2001:420:db8:11::444:d1]:10250/containerLogs/default/my-nginx-deploy-64fdd68f8d-klg78/nginx": Gateway Timeout

Be great, if someone helps me resolve this.

Additional Information:

1. When running the kubectl command with -v=9

I0122 06:15:39.727678   23430 round_trippers.go:435] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.21.0 (linux/amd64) kubernetes/cb303e6" 'https://10.78.109.207:6443/api/v1/namespaces/default/pods/my-nginx-deploy-64fdd68f8d-klg78/log'
I0122 06:15:39.734842   23430 round_trippers.go:454] GET https://10.10.10.207:6443/api/v1/namespaces/default/pods/my-nginx-deploy-64fdd68f8d-klg78/log 500 Internal Server Error in 7 milliseconds
I0122 06:15:39.734855   23430 round_trippers.go:460] Response Headers:
I0122 06:15:39.734862   23430 round_trippers.go:463]     Cache-Control: no-cache, private
I0122 06:15:39.734868   23430 round_trippers.go:463]     Content-Type: application/json
I0122 06:15:39.734873   23430 round_trippers.go:463]     Content-Length: 219
I0122 06:15:39.734879   23430 round_trippers.go:463]     Date: Sat, 22 Jan 2022 11:15:39 GMT
I0122 06:15:39.734893   23430 request.go:1123] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \"https://[2001:420:db8:11::444:d1]:10250/containerLogs/default/my-nginx-deploy-64fdd68f8d-klg78/nginx\": Gateway Timeout","code":500}
I0122 06:15:39.735207   23430 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://[2001:420:db8:11::444:d1]:10250/containerLogs/default/my-nginx-deploy-64fdd68f8d-klg78/nginx\": Gateway Timeout",
  "code": 500
}]
F0122 06:15:39.735240   23430 helpers.go:115] Error from server: Get "https://[2001:420:db8:11::444:d1]:10250/containerLogs/default/my-nginx-deploy-64fdd68f8d-klg78/nginx": Gateway Timeout

2. kubelet is listening on IPv6.

netstat -tulpn | grep kubelet
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      2734333/kubelet     
tcp        0      0 127.0.0.1:3507          0.0.0.0:*               LISTEN      2734333/kubelet     
tcp6       0      0 :::10250                :::*                    LISTEN      2734333/kubelet

3. I try to simulate the HTTP request as probably performed by kube-apiserver and I am getting success (as shown below). But, not sure, why kubectl logs podname command is failing.

curl -k -g https://[2001:420:db8:11::444:d1]:10250/containerLogs/default/my-nginx-deploy-64fdd68f8d-klg78/nginx --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt --key /etc/kubernetes/pki/apiserver-kubelet-client.key 
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/01/21 11:46:44 [notice] 1#1: using the "epoll" event method
2022/01/21 11:46:44 [notice] 1#1: nginx/1.21.5
2022/01/21 11:46:44 [notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6) 
2022/01/21 11:46:44 [notice] 1#1: OS: Linux 3.10.0-1062.1.2.el7.x86_64
2022/01/21 11:46:44 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2022/01/21 11:46:44 [notice] 1#1: start worker processes
2022/01/21 11:46:44 [notice] 1#1: start worker process 31
2022/01/21 11:46:44 [notice] 1#1: start worker process 32
2022/01/21 11:46:44 [notice] 1#1: start worker process 33
2022/01/21 11:46:44 [notice] 1#1: start worker process 34`

4. When I run kubelet with -v=8 and run send the HTTP request manually from master node (as shown in #3 above), I can see the request is received by kubelet (whose IPv6 address is 2001:420:db8:11::444:cf).

journalctl -u kubelet.service -f | grep -E "/var/lib/docker/containers|srcIP"
Jan 22 06:43:46 xyz-k8s-33 kubelet[2734333]: I0122 06:43:46.471490 2734333 logs.go:319] "Finished parsing log file" path="/var/lib/docker/containers/72a362c92adbe1315c9065cd2d2148cb775412f50edcd52dc21ed660e73209da/72a362c92adbe1315c9065cd2d2148cb775412f50edcd52dc21ed660e73209da-json.log"
Jan 22 06:43:46 xyz-k8s-33 kubelet[2734333]: I0122 06:43:46.471577 2734333 httplog.go:89] "HTTP" verb="GET" URI="/containerLogs/default/my-nginx-deploy-64fdd68f8d-klg78/nginx" latency="14.793967ms" userAgent="curl/7.29.0" srcIP="[2001:420:db8:11::444:cf]:49373" resp=200

5. But when I run the command kubectl logs podname, I do not see such logs on kubelet. This makes me feel, kube-apiserver did not sent the request at all??? It simply failed to do so.