service "kubernetes" deleted - accidentally deleted kubernetes service

6/13/2019

I accidentally deleted the kubernetes svc:

service "kubernetes" deleted

using:

 kubectl delete svc --all

what should I do? I was just trying to remove services so I could launch new ones.

--
amazon-eks
eks
kubernetes

1 Answer

6/17/2019

A bit theory first ;) Whenever you delete kubernetes svc, you also delete endpoint and this is where Reconciler comes in. It is actually a controller manager for the core bootstrap Kubernetes controller loops, which manage creating the "kubernetes" service, the "default", "kube-system" and "kube-public" namespaces, and provide the IP repair check on service IPs.

So, in healthy clusters default.kubernetes service should be automatically recreated by controller manager.

If it's not, I'd recommend to:

Check api-server logs

kubectl logs -f kube-apiserver-master -n kube-system

You should see something like:

Resetting endpoints for master service "kubernetes" to [10.156.0.3]

If you don't see it, try to manually remove etcd key for this service

Because the current state of the cluster is stored in etcd, it may happen that the key remain when you deleted a service:

a. exec to etcd-master pods

kubectl exec -it etcd-master -n kube-system sh

b. get the etcd key value

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt get /registry/services/endpoints/default/kubernetes

c. if you get any value like:

v1    Endpointst
O

kubernetesdefault"*$eafc04cf-90f3-11e9-a75e-42010a9c00032����z!


10.156.0.3
https�2TCP"

just remove it by

ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --key=/etc/kubernetes/pki/etcd/server.key --cert=/etc/kubernetes/pki/etcd/server.crt rm /registry/services/endpoints/default/kubernetes

After you did it, check the api-server logs once again.

-- A_Suh
Source: StackOverflow