I have an 7.4.0 ES cluster using ECK 1.0 and after my 3 dedicated master nodes got out of disk space, I deleted them along with the volumes to test a critical scenario.
Once the new eligible masters were created, they couldn't elect a new member. Now the cluster is stuck forever although it sees the new master eligible servers (pods in k8s).
Is there a way to force ES to elect a new master even though the previous ones are out of the picture?
Be aware that the masters had no data. All the data resides on data only nodes. Unfortunately, I cannot access them as long as a master is not elected.
Having no snapshot in place yet, I resolved myself around bringing the data nodes into a new cluster.
I recreated the 3 master nodes and made sure they joined a new cluster with an elected master. Following that, I detached the data nodes from their previous cluster and let them associate with the new one. I knew the risk and fortunately, I didn't lose data except for my xpack security credentials.
I followed the procedure here and adapted it for my ECK case by updating the init container of the statefulset.
After such pain, I cannot stress enough the value of snapshots.
Be aware that the masters had no data.
This is not really true. The master nodes hold the cluster metadata which Elasticsearch needs to correctly understand the data stored on the data nodes. Since you've deleted the metadata, the data on the data nodes is effectively meaningless.
At this point your best option is to start again with a new cluster and restore your data from a recent snapshot.