We plan to set up a ES cluster using Kubernetes. The k8s pod will consist of 3 data nodes and we intend to have persistent volume mapping for the data.
I am new to both ES and k8s but as I understand, in a pod, a volume is usually shared. But in this case, we should not share the data of the 3 instances. What is the best way to achieve volume mapping for a pod. If I have 3 instances, should I map each separately like /node1/data, /node2/data or should I just map to a single volume '/data' and ES will ensure that that data between the instances is isolated?
I am not sure if mapped volume should have node names like node1, node2 etc as explained above. Is the pod design of having 3 ES nodes (or multiple data nodes) correct? Can you specify different persistent volumes for different node instances in a pod?
What happens if the pod is terminated and a new pod is created? Should there be a strong binding between the mapped volume and nodename. What is the best practice to achieve this?
For starters, to some extent it depends on how your cluster is provisioned/run. If you run on a decent cloud provider, kube will be able to support you with automatic creation of PV for your PVC (Persistent Volumes).
Did I understand correctly that you plan to have 3 nodes inside one pod? That does not sound right. You should have 3 separate pods forming your cluster. As this is a stateful service, you might want to look into StatefulSet
Having correctly defined PVC in your cluster pods, kube will make sure that they are correctly bound to your pods (ie. on AWS AZ always schedule node in correct zone so that it can link that pods EBS based PV).
I would suggest you to start building the cluster using emptyDir volume in your pod definition, and when you get a hang of it, you can move on to making sure that your data is correctly persisted when pods are deleted and recreated with update of your Deployment or StatefulSet.
Sidenote: while inside pod singe volume can be mounted in multiple containers of that pod, not many backends support ReadWriteMany access mode allowing mounting the same volume to different pods (ie. NFS does)