Unable to run pods on new node

6/28/2019

Had to change node (server) with the new one leaving the same node name. What I did was:

  • master> kubectl delete no srv1 (removing old node)
  • srv1> kubeadm join... (joining new node)

after new node joined cluster no pods can be created.

Warning  FailedCreatePodSandBox  16s               kubelet, srv1  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b85728b51a18533e9d57f6a1b1808dbb5ad72bff4d516217de04e7dad4ce358d" network for pod "dpl-6f56777485-6jzm6": NetworkPlugin cni failed to set up pod "dpl-6f56777485-6jzm6_default" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.16.1/24
-- Jonas
kubernetes

1 Answer

7/3/2019

Ideally when performing such a task like "replacing a node" below steps should be considered:

  1. Drain node kubectl drain NODE_NAME
  2. Reset that node kubeadm reset in the old node (optional step if the old node is accessible)
  3. Finally kubeadm delete node NODE_NAME

Things to consider when replacing a old node with new node:

  1. The new node should have the same name as the old node which is echo $HOSTNAME should remain same.
  2. The new node should have the same ip as the old one.

Because these are a node identity.

Finally in a scenario where you have already performed kubectl delete node ... and replaced it with a new one.

curl -LO https://raw.githubusercontent.com/coreos/flannel/62e44c867a2846fefb68bd5f178daf4da3095ccb/Documentation/kube-flannel.yml
kubectl delete -f kube-flannel.yml

[perform below in the nodes which are having problems]

sudo ip link del cni0
sudo ip link del flannel.1
sudo systemctl restart network

[re-apply network plugin]

kubectl apply -f kube-flannel.yml
-- garlicFrancium
Source: StackOverflow