Elasticsearch helm chart pod stuck in "Init:CrashLoopBackOff" state

1/24/2019

I am deploying the helm chart for Elastic-stack on a bare-metal k8s cluster here https://github.com/helm/charts/tree/master/stable/elastic-stack

This includes the helm chart for Elasticsearch here https://github.com/helm/charts/tree/master/stable/elasticsearch

The Elastic-stack chart calls the Elasticsearch with the default values in values.yaml, and I am not setting anything else.

After helm installing, I see the pods elastic-stack-elasticsearch-data-0 and elastic-stack-elasticsearch-master-0 are stuck in Init:CrashLoopBackOff (after repeating Init:Error for some time).

kubectl describe pod shows me that the problem is with the initContainer called chown. The code for this container is here https://github.com/helm/charts/blob/master/stable/elasticsearch/templates/data-statefulset.yaml#L79

The relevant output from describe pod is not very helpful:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 24 Jan 2019 05:35:14 +0000
  Finished:     Thu, 24 Jan 2019 05:35:14 +0000
Ready:          False
Restart Count:  1
Environment:    <none>
Mounts:
  /usr/share/elasticsearch/data from data (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from elastic-stack-elasticsearch-data-token-kgtsm (ro)

I know that it is able to pull the image, since it pulled it for the previous initContainer.

I have a feeling it has to do with the Mounts. What exactly is /usr/share/elasticsearch/data from data (rw) doing? I created persistentvolumes called es-data-volume and es-master-volume and they have been claimed by data-elastic-stack-elasticsearch-data-0 and data-elastic-stack-elasticsearch-master-0. Is that line looking for a volume named data?

I don't know where to look to troubleshoot this problem. What could be some possible causes of this issue?

-- swagrov
elasticsearch
kubernetes
kubernetes-helm

1 Answer

2/17/2019

I had the same issue and this has fixed for me, I changed settings on my NFS server (sudo vim /etc/exports)

from:

/data/nfs/kubernetes 192.168.1.0/24(rw,sync,no_subtree_check)

to :

/data/nfs/kubernetes 192.168.1.0/24(rw,insecure,sync,no_subtree_check,no_root_squash)

from what I understood no_root_squash is the key

Hopefully that will solve it for you as well

-- Marcin Gorzynski
Source: StackOverflow