We are currently using the bitnami charts (https://github.com/bitnami/charts) to deploy Elastic to AKS. It works pretty well, ultimately it create 3 master nodes and 2 data nodes.

I was trying to accomplish data persistence so if someone destroys the cluster and we need to rebuild it, we can simply hook up the storage and re-deploy. I am quickly finding that this might not actually work as I was hoping.

Let me explain what I did before I ask the question.

I created the elastic install for the first time without issue. If I bounce the pod, it reads the pv without issue and my data is still present. If I delete the entire cluster but keep the azure file, I re-setup the exact same PV and the PVC binds correctly, but I get the following error.

Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: This node previously joined a cluster with UUID GBu9aCIYQy23TyA_JKQM6Q and is now trying to join a different cluster with UUID po3rwsyhRTyKmN3Z_luhkQ. This is forbidden and usually indicates an incorrect discovery or cluster bootstrapping configuration. Note that the cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path /bitnami/elasticsearch/data which will also remove any data held by this node.

So ultimately I am wondering if what I am trying to accomplish is even doable. Bottom line, I was hoping that I could have a storage account (azure file) which allowed me to just re-build my cluster with elastic ay any given time. Even in a different RG if I wanted to. We use Terraform, so I am trying to ensure that if the cluster needs to be rebuilt that I have the ability to get back with the persistent storage. Is what we are trying to do possible and whats the recommended approach if so?

TIA