I am trying to create Kubernetes V1.11.0 HA cluster in Centos 7 server. I have IPV6 disabled on my host.
https://kubernetes.io/docs/setup/independent/high-availability/
Based on these steps added all three master, coreDNS POD is not coming up, its keep logging below error in /var/log/messages file.
kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcdf6894-8t7bt 0/1 ContainerCreating 0 53m
coredns-78fcdf6894-plmll 0/1 ContainerCreating 0 53merror message:
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531244 10041 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531286 10041 kuberuntime_sandbox.go:56] CreatePodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531298 10041 kuberuntime_manager.go:646] createPodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531358 10041 pod_workers.go:186] Error syncing pod 7caa3f29-89c3-11e8-aa0f-00505693ca30 ("coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)"), skipping: failed to "CreatePodSandbox" for "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)\" failed: rpc error: code = Unknown desc = [failed to set up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to set up pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to teardown pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: failed to get IP addresses for \"eth0\": <nil>]"
Jul 17 10:09:28 master03 kubelet: W0717 10:09:28.636173 10041 cni.go:243] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "88183d8a24dcafca5b04e3a6454e83aca1417a6934f50cd52b3e6fffbae4f272"
Jul 17 10:09:28 master03 kernel: XFS (dm-18): Mounting V4 Filesystem
How to fix this error message and bring the coreDNS PODs?
Thanks
There is an issue on Coreos Github that describes exactly your case. And the suggestion to solve the issue was:
squeed: You are using an old version of CNI, which does not work on systems with ipv6 disabled. Please re-enable ipv6 or upgrade CNI.
And regarding the related issue on CNI Github he also mentioned:
squeed: As I already explained, this has been fixed for 6 months. Please upgrade your CNI plugins.
And jellonek commented:
This is the question for your deployment software provider, which in your case is probably https://github.com/kubernetes/kubeadm
However, in cases when you need to turn off ipv6 for some reason, you can install a cluster with addon KubeDNS instead of CoreDNS using kubeadm:
kubeadm init --pod-network-cidr=192.168.0.0/16 --feature-gates=CoreDNS=false
Or you can delete CoreDNS deployment and service, and then apply KubeDNS yaml, which should be generated or adjusted to your actual cluster configuration.
Here is a function from the deployAddons.sh file responsible for deploying kube-dns addon:
function deploy_dns {
echo "Deploying DNS on Kubernetes"
cp "${KUBE_ROOT}/cluster/addons/dns/kube-dns/kube-dns.yaml.sed" kube-dns.yaml
sed -i -e "s/\\\$DNS_DOMAIN/${DNS_DOMAIN}/g" kube-dns.yaml
sed -i -e "s/\\\$DNS_SERVER_IP/${DNS_SERVER_IP}/g" kube-dns.yaml
KUBEDNS=`eval "${KUBECTL} get services --namespace=kube-system | grep kube-dns | cat"`
if [ ! "$KUBEDNS" ]; then
# use kubectl to create kube-dns addon
${KUBECTL} --namespace=kube-system create -f kube-dns.yaml
echo "Kube-dns addon is successfully deployed."
else
echo "Kube-dns addon is already deployed. Skipping."
fi
echo
}
I fixed this with the reference of these links
check with kubernates troubleshooting guide and the networking plugin addons you have used
https://kubernetes.io/docs/concepts/cluster-administration/addons/
And install the following with those and check
kubectl get pods -n kube-systemI have to enable the IPv6 to my CoreDNS pod to run.
Here is the steps I followed:
https://www.thegeekdiary.com/how-to-enable-ipv6-on-centos-rhel-7/
Edit /etc/default/grub and change the value of kernel parameter ipv6.disable from 1 to 0 in line:
# grub2-mkconfig -o /boot/grub2/grub.cfg
# shutdown -r nowI solved the problem without IPv6 enabling (with help from https://www.facebook.com/groups/k8skr)
So, the reason is that, default Kubernetes CNI of Ubuntu is 0.6.0. But the problem is fixed with Kubernetes CNI 0.7.0 So you can update it through download from the below site and replace bin files to /opt/cni/bin/
https://github.com/containernetworking/plugins/releases/tag/v0.7.1
At least, it works for me :)