We have a 6 node K8s kubernetes cluster with Rancher running on AWS . The cluster was shutdown and brought up, cluster came back up fine but it has gone back to old state and the few new namespaces are missing , is there a way to recover the namespaces and the pods related to them.
Thanks,
If you have used Kops for Kubernetes cluster implementation on AWS, then it might be possible to restore last state of your cluster from S3 bucket, where Kops stores its state as well, by specifying export KOPS_STATE_STORE=s3://yourstatestore environment variable, look at this link to get more information.
However, keep in mind that Kops can revert back only general configuration of your Kubernetes cluster, thus the internal resources (Deployments, Pods, Namespaces, etc.) are not included to this restore list.
Therefore you can consider to use etcd datastore backup. For example, using etcdctl command line tool and regularly create snapshots of etcd data store.
Also, you can think about using heptio/ark utility for managing disaster recovery events, specifically for Kubernetes cluster resources and persistent volumes.