I am trying to setup RabbitMQ Cluster on K8S Cluster (CentOS 7) but it errored with
2021-07-01 16:30:10.048 [error] <0.273.0> Failed to fetch a list of nodes from Kubernetes API: {failed_connect,[{to_address,{"kubernetes.default",443}}, {inet,[inet],nxdomain}]}
I tried debugging the issue but I am not sure what's missing
nslookup output
[root@re-devk8s-control01 tmp]# nslookup kubernetes.default
Server: 174.25.1.14
Address: 174.25.1.14#53
** server can't find kubernetes.default: NXDOMAIN
dnsutils output
[root@re-devk8s-control01 tmp]# kubectl exec -i -t dnsutils -- cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local awstruewind.local
options ndots:5
/etc/resolv.conf
[root@re-devk8s-control01 tmp]# cat /etc/resolv.conf
# Generated by NetworkManager
search ourdomain.local
nameserver 174.25.1.14
KubeDNS log and I see few errors in it
[root@re-devk8s-control01 tmp]# kubectl logs --namespace=kube-system -l k8s-app=kube-dns
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.0
linux/amd64, go1.15.3, 054c9ae
[ERROR] plugin/errors: 2 610092700101479936.1385205460227937641. HINFO: read udp 10.244.0.31:47645->174.25.1.14:53: read: no route to host
[ERROR] plugin/errors: 2 610092700101479936.1385205460227937641. HINFO: read udp 10.244.0.31:60675->174.25.1.14:53: read: no route to host
Though there were many similar posts in SO, I was unable to find out the root cause
I found the following link from SO which helped me to resolve the issue KubeDNS Masquerade
firewall-cmd --list-all
...
masquerade: yes
Enable if it's "no":
firewall-cmd --add-masquerade --permanent
firewall-cmd --reload