K
Q

Best kubernetes storage options for large files

3/6/2019

I've been looking into kubernetes for docker orchestration and one of my use cases is to have multiple containers spawned on different nodes, and each container needs to have read access to a list of very large files (20G+).

Because the files can be updated at times, we will be use block volume. I'm running the cluster on esxi we are limited to open source and non-cloud solutions...

By reading https://kubernetes.io/docs/concepts/storage/volumes/, seems like one of the options would be portworx. I'm wondering if there's any other better options out there?

I assume this is a pretty common use case. I'm very new to Kubernetes so any help/advice will be greatly appreciated!

-- Kevin Yu
docker
esxi
kubernetes
storage

Similar Questions

k8s - IP and DNS for postgres with service
installing helm chart stable/mssql-linux I got "pod has unbound PersistentVolumeClaims"
Change Kubernetes Instance Template to open HTTPS port
Kubernetes showing Image Pull Error while scheduling pods
what's the meaning of apiVersion of batch/v1beta1
How to create a service account to get a list of pods from inside a Kubernetes cluster?

2 Answers

3/6/2019

look at one of the below storage solution for your cluster

  1. glusterfs
  2. cephfs
  3. nfs ( can be used for dev and test and not for prod )
-- P Ekambaram
Source: StackOverflow
3/6/2019

First of all on Volumes, there are so many options also depends upon where your cluster is hosted (on prem or managed cloud provider?) managed cloud providers usually would have either their own option of easy mounting their block storage options e.g. Azure storage by Azure, S3 for AWS OR third party driver solutions and so on but things to know and note here:

  • On Kube volumes, you mentioned - though its outlives container but within Pod and when a Pod ceases to exist, the volume will cease to exist, too. which means you can not use across nodes for diff Pods acorss nodes . This has to be used within specific Pod
  • In case options available, consider managed cloud provider's default block mount solutions. It would be less painful for integration and also persistence issues
  • Finally, from design perspective, this seems against microservice/docker/containerization pattern - you may want to revisit your original need and goals as Pods are supposed to be created on the fly and stateless as possible - created and recreated whenever needed if any issues or for scalability need

Hope this helps

-- AnilR
Source: StackOverflow