I have tried to deploy so far in the below 2 ways:
1. Deploy it as a statefulset on Kubernetes and using persistent
volume of nfs as its storage . But then I learnt that we shouldn't
be running minio backed by NAS as erasure code should not be run on NAS data.
2. Deploy it as Daemonset using local volumes. This time I attached
separate disks to my nodes and labelled them such
that minio is scheduled to run on these nodes. The disks are mounted on
/data/minio on each of the nodesBut now everyday, the nodes are coming under disk pressure and minio pods are getting evicted. When I check the kubelet logs:
Aug 13 21:05:45 staging-node2 kubelet[2188]: I0813 21:05:45.968179 2188 kubelet_pods.go:1073] Killing unwanted pod "minio-kjrkc"
Aug 13 21:05:45 staging-node2 kubelet[2188]: I0813 21:05:45.975372 2188 kuberuntime_container.go:559] Killing container "docker://6da1247718f8e6c92399e231f8c31ff1c510737c658ac2aca87c1659aa6b51cc" with 30 second grace periodIt tries to kill the pods but the container never dies. Even if minio gets TERMINATED signal, container is still up.
What other options are left for an on-prem minio installations?
Is using local storage somehow not recommended or am I using it incorrectly ?
Any idea if I have to explicitly configure any pre-hook for minio to accept the terminate signal ?
Right approach to deploy MinIO on Kubernetes is using local volumes on multiple distributed nodes using a StatefulSet. You can do this via Yaml files or Helm Chart or via our MinIO Operator. Docs are available here https://github.com/minio/minio/tree/master/docs/orchestration/kubernetes
What other options are left for an on-prem minio installations?
A StatefulSet with atleast 4 Pods and storage via hostPath volumes is generally the right way to deploy. Depending on the use case and existing infrastructure there may be other relevant approaches too.
Is using local storage somehow not recommended or am I using it incorrectly ?
Local storage is recommended. I'll need more info to see what exactly is going wrong here, but if you have dedicated local storage drives, you should be fine.
Any idea if I have to explicitly configure any pre-hook for minio to accept the terminate signal ?
MinIO handles signals to terminate etc. There is no pre-hook needed as such.
Hope this helps, please join us on our Slack channel @ https://slack.min.io for detailed discussion etc.