I am not able to curl/wget any URL with https. They're all giving connection refused errors. Here's what I've observed so far:
When I curl any URL with https from any pod, the domain gets resolved to the different IP address than the intended one. I verified this with dig domainname
and curl
ing the same domainname. Both IP's were different
For debugging purpose, I tried the same scenario from a kubelet docker container and it worked. But if I tried the same from another app container, it fails.
Any idea what might be wrong? I am sure, there is some issue with networking. Any more steps for debugging?
The cluster is setup with RKE on bare-metal which uses canal for networking.
The website I am trying to curl is updates.jenkins.io and here's the nslookup output
bash-4.4# nslookup updates.jenkins.io
Server: 10.43.0.10
Address: 10.43.0.10#53
Non-authoritative answer:
updates.jenkins.io.domain.name canonical name = io.domain.name.
Name: io.domain.name
Address: 185.82.212.199
And nslookup from the node gives
root@n4:/home# nslookup updates.jenkins.io
Server: 127.0.1.1
Address: 127.0.1.1#53
Non-authoritative answer:
updates.jenkins.io canonical name = mirrors.jenkins.io.
Name: mirrors.jenkins.io
Address: 52.202.51.185
As far as I can see, it is trying to connect to io.domain.name and not updates.jenkins.io.
Further inspection, all domains ending with .io are causing the issue. Here'a another one:
bash-4.4# nslookup test.io
Server: 10.43.0.10
Address: 10.43.0.10#53
Non-authoritative answer:
test.io.domain.name canonical name = io.domain.name.
Name: io.domain.name
Address: 185.82.212.199
Well, there was some issue with /etc/resolv.conf
. It was missing the correct nameserver entry. Once that was resolved, and the system components were restarted, everything was working.