I have a database running in a Kubernetes cluster on AWS. The database is deployed as a StatefulSet with 3 replicas. Each replica uses an AWS EBS storage as its persistent volume.
If I shutdown a database node, Kubernetes starts automatically a new one. The newly started node finds its corresponding persistent volume (AWS EBS volume) without any problems.
But what happens if I shut the Kubernetes cluster down? The AWS EBS volumes are still there. But does the Kubernetes cluster or the database StatefulSet find its corresponding persistent volumes on AWS after a full cluster restart?
Kubernetes relies on etcd for state storage. If you're using kops to bring your cluster up then your etcd is backed up by AWS EBS volumes. It is recommended to backup your etcd periodically to be able to fully recover from disaster.
See here: https://github.com/kubernetes/kops/blob/master/docs/etcd_backup.md