K
Q

My kubernetes cluster IP address changed and now kubectl will no longer connect

10/4/2018
  • Running under Ubuntu I used kubeadm init to setup my cluster (master node) and copied over the /etc/kubernetes/admin.conf $HOME/.kube/config and all was well when using kubectl.
  • However after a reboot my master node has had an IP address change which is not the same as what is in $HOME/.kube/config so now I can no longer connect kubectl

So how do I regenerate the admin.conf now that I have a new IP address? Running kubeadm init will just kill everything which is not what I want.

-- TheEdge
kubernetes

Similar Questions

How to get which user has created a revision of a resource?
HorizontalPodAutoscaler multiple replica upscale
CoreDNS fails to run in Kubernetes cluster
kubernetes-dashboard exposing through istio [1.0.0] ingress --istio-ingressgateway
NoClassDefFoundError and DuplicateInstanceNameException for hazelcast in kubernetes
Azure Kubernetes Service: How to move the pods from the Spot node pool to Regular node pool after the Spot node pool is evicted?

3 Answers

10/4/2018

The following command can be used to regenerate admin.conf

kubeadm alpha phase kubeconfig admin --apiserver-advertise-address <new_ip>

However, if you use an IP instead of a hostname, your API-server certificate will be invalid. So, either regenerate your certs ( kubeadm alpha phase certs renew apiserver ), use hostnames instead of IPs or add the insecure --insecure-skip-tls-verify flag when using kubectl

-- rat_salad
Source: StackOverflow
10/4/2018

You do not want to use kubeadm reset. That will reset everything and you would have to start configuring your cluster again.

Well, in your scenario, please have a look on the steps below:

  1. nano /etc/hosts (update your new IP against YOUR_HOSTNAME)

  2. nano /etc/kubernetes/config (configuration settings related to your cluster) here in this file look for the following params and update accordingly

    KUBE_MASTER="--master=http://YOUR_HOSTNAME:8080"

    KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME:2379" #2379 is default port

  3. nano /etc/etcd/etcd.conf (conf related to etcd)

    KUBE_ETCD_SERVERS="--etcd-servers=http://YOUR_HOSTNAME/WHERE_EVER_ETCD_HOSTED:2379"

    2379 is default port for etcd. and you can have multiple etcd servers defined here comma separated

  4. Restart kubelet, apiserver, etcd services.

It is good to use hostname instead of IP to avoid such scenarios.

Hope it helps!

-- umairali
Source: StackOverflow
1/16/2020

I found this solution on the internet and it works for me:

    systemctl stop kubelet docker
    cd /etc/
    mv kubernetes kubernetes-backup
    mv /var/lib/kubelet /var/lib/kubelet-backup
    mkdir -p kubernetes
    cp -r kubernetes-backup/pki kubernetes
    rm kubernetes/pki/{apiserver.*,etcd/peer.*}
    systemctl start docker
    kubeadm init --ignore-preflight-errors=DirAvailable--var-lib-etcd
    #Run "kubeadm reset" on all nodes if was this error "error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR FileAvailable--etc-kubernetes-kubelet.conf]: /etc/kubernetes/kubelet.conf already exists
        [ERROR Port-10250]: Port 10250 is in use
        [ERROR FileAvailable--etc-kubernetes-pki-ca.crt]: /etc/kubernetes/pki/ca.crt already exists"
    cp kubernetes/admin.conf ~/.kube/config
    kubectl get nodes --sort-by=.metadata.creationTimestamp
    kubectl delete node $(kubectl get nodes -o jsonpath='{.items[(@.status.conditions[0].status=="Unknown")].metadata.name}')
    kubectl get pods --all-namespaces

After These, Join your Slaves to Master. Reference: https://medium.com/@juniarto.samsudin/ip-address-changes-in-kubernetes-master-node-11527b867e88

-- Amir Soleimani
Source: StackOverflow