I've been trying to get Elasticsearch running on K8s using the newly-promoted-to-stable helm chart, which works fine, BTW, for elasticsearch v 6.4.2. However, we're tied to a grails app that requires elasticsearch v 5.5.3, for which we don't have the ability to upgrade. I've downgraded the elasticsearch image version in the chart to 5.5.3 (and also tried v 5.6.12) but it fails to start.
I looked into the elasticsearch operator, but it's currently set up to work with AWS S3 storage types, out of the box, and GCP with a little work (although no snapshot ability). Before I dive into this, I'd like to know if it will work with 5.5.3, to begin with.
Does anyone know if I can get elasticsearch v 5.5.3 running in a k8s cluster? I would say using a k8s StatefulSet at a minimum.
Thanks!
Update
I suppose I should have given the errors that the existing helm chart is having when downgrading elasticsearch image to 5.5.3.
master-0 pod fails to start with:
Error injecting constructor, ElasticsearchException[java.io.IOException: failed to read [id:15, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/global-15.st]]; nested: IOException[failed to read [id:15, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/global-15.st]]; nested: ElasticsearchException[Unknown license version found, please upgrade all nodes to the latest elasticsearch-license plugin]; at org.elasticsearch.gateway.GatewayMetaState.<init>(Unknown Source) while locating org.elasticsearch.gateway.GatewayMetaState for parameter 4 at org.elasticsearch.gateway.GatewayService.<init>(Unknown Source) while locating org.elasticsearch.gateway.GatewayService Caused by: ElasticsearchException[java.io.IOException: failed to read [id:15, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/global-15.st]]; nested: IOException[failed to read [id:15, legacy:false, file:/usr/share/elasticsearch/data/nodes/0/_state/global-15.st]]; nested: ElasticsearchException[Unknown license version found, please upgrade all nodes to the latest elasticsearch-license plugin];
The client pods fail with:
[2018-10-22T17:52:51,835][WARN ][o.e.d.z.UnicastZenPing ] [elasticsearch-client-6bf954c595-7zlpc] failed to resolve host [elasticsearch-discovery] java.net.UnknownHostException: elasticsearch-discovery
Clearly, it's expecting a later elasticsearch version.
We ended up going with a variation of this Medium post to get ES 5.5.3 working in "multi-node" mode. This is not HA in the way the latest ES chart version provides but will get us on our way.
We used a regular storage class, instead of the SSD and we had to disable ES boot checking, for now, because we couldn't get the ulimit command to work on ES init.
We have multiple ES nodes and are good until we can upgrade the app to use ES latest version.
The short answer here is that ElasticSearch 5.5.3 should work with Kubernetes. Note the configs for 5.5.3 are slightly different, I believe they changed after 5.6 where for example the made x-pack enabled by default, and yes use a StatefulSet or start from the existing stable Helm chart.