GEK does not automatically start new node after I delete a node from the GKE cluster

8/15/2018

I created a cluster:

gcloud container clusters create test

so there will be 3 nodes:

kubectl get nodes
NAME                                 STATUS    ROLES     AGE       VERSION
gke-test-default-pool-cec920a8-9cgz   Ready     <none>    23h       v1.9.7-gke.5
gke-test-default-pool-cec920a8-nh0s   Ready     <none>    23h       v1.9.7-gke.5
gke-test-default-pool-cec920a8-q83b   Ready     <none>    23h       v1.9.7-gke.5

then I delete a node from the cluster

kubectl delete node gke-test-default-pool-cec920a8-9cgz
node "gke-test-default-pool-cec920a8-9cgz" deleted

no new node is created.

Then I delete all nodes. still there is no new node created.

kubectl get nodes
No resources found.

Am I doing something wrong? I suppose it can automatically bring up new node if some node died.

-- gacopu
google-cloud-platform
google-kubernetes-engine
kubernetes

2 Answers

8/20/2018

Kubernetes is a system for managing workloads and not the machines. Kubernetes node object reflects the state of the underlying infrastructure.

As such node objects are automatically managed by Kubernetes. "kubectl delete node" simply removes a serialized object from Kubernetes "etcd" storage. It does nothing to delete VM on GCE side where the kubernetes node is hosted. "kubectl delete node" is not meant to be used to remove nodes. Node pool itself carries the desired declared state, which cannot be altered by the "kubectl delete node" command.

If you want to remove a node you should resize the instance group.

-- Rahi
Source: StackOverflow

8/22/2018

After running kubectl delete node gke-test-default-pool-cec920a8-9cgz run gcloud compute instances delete gke-test-default-pool-cec920a8-9cgz

This will actually delete VM (kubectl delete only "disconnects" it from the cluster). GCP will recreate the VM and it will automatically rejoin the cluster.

-- Maciek Sawicki
Source: StackOverflow