Accidentally deleted Kubernetes namespace

10/14/2019

I have a Kubernetes cluster on google cloud. I accidentally deleted a namespace which had a few pods running in it. Luckily, the pods are still running, but the namespace is in terminations state.

Is there a way to restore it back to active state? If not, what would the fate of my pods running in this namespace be?

Thanks

-- Shadman Anwer
google-cloud-platform
kubernetes

2 Answers

1/20/2020

backup most resource configuration reguarly:

kubectl get all --all-namespaces -o yaml > all-deploy-resources.yaml

but this is not includes all resources.

another ways

by ark/velero:

https://github.com/vmware-tanzu/velero (Backup and migrate Kubernetes applications and their persistent volumes https://velero.io)

-- BMW
Source: StackOverflow

10/14/2019

A few interesting articles about backing up and restoring Kubernetes cluster using various tools:

https://medium.com/@pmvk/kubernetes-backups-and-recovery-efc33180e89d

https://blog.kubernauts.io/backup-and-restore-of-kubernetes-applications-using-heptios-velero-with-restic-and-rook-ceph-as-2e8df15b1487

https://www.digitalocean.com/community/tutorials/how-to-back-up-and-restore-a-kubernetes-cluster-on-digitalocean-using-heptio-ark

https://www.revolgy.com/blog/kubernetes-in-production-snapshotting-cluster-state

I guess they may be useful rather in future than in your current situation. If you don't have any backup, unfortunately there isn't much you can do.

Please notice that in all of those articles they use namespace deletion to simulate disaster scenario so you can imagine what are the consequences of such operation. However the results may not be seen immediately and you may see your pods running for some time but eventually namespace deletion removes all kubernetes cluster resources in a given namespace including LoadBalancers or PersistentVolumes. It may take some time. Some resource may not be deleted because it is still used by another resource (e.g. PersistentVolume by running Pod).

You can try and run this script to dump all your resources that are still available to yaml files however some modification may be needed as you will not be able to list objects belonging to deleted namespace anymore. You may need to add --all-namespaces flag to list them.

You may also try to dump any resource which is still available manually. If you still can see some resources like Pods, Deployments etc. and you can run on them kubectl get you may try to save their definition to a yaml file:

kubectl get deployment nginx-deployment -o yaml > deployment_backup.yaml

Once you have your resources backed up you should be able to recreate your cluster more easily.

-- mario
Source: StackOverflow