K
Q

elasticsearch on kubernetes - discovery of nodes

9/24/2015

We are attempting to run Elasticsearch on top of a kubernetes / flannel / coreos cluster.

As flannel does not support multicast, we cannot use Zen multicast discovery to allow the nodes to find each other, form a cluster and communicate.

Short of hard-coding the IP addresses of all the kubernetes nodes into the ES-config-file, is there another method we can utilise to assist in discovery? Possibly using etcd2 or some other kubernetes-compatible discovery service?

-- DrGecko
consul
elasticsearch
kubernetes

Similar Questions

Following Node.js hello-world tutorial, cannot delete cluster
App container to cassandra node - one to one or?
How to get the real ip in the request in the pod in kubernetes
Can't resolve 'kubernetes' by skydns serivce in Kubernetes

3 Answers

10/5/2015

There is a discovery plugin that uses the kubernetes API for cluster discovery:

https://github.com/fabric8io/elasticsearch-cloud-kubernetes

Install the plugin:

/usr/share/elasticsearch/bin/plugin -i io.fabric8/elasticsearch-cloud-kubernetes/1.3.0 --verbose

Create a Kubernetes service for discovery:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-cluster
spec:
  ports:
    - port: 9300
  selector:
    app: elasticsearch

And an elasticsearch.yml:

cloud.k8s.servicedns: elasticsearch-cluster
discovery.type: io.fabric8.elasticsearch.discovery.k8s.K8sDiscoveryModule
-- kjv
Source: StackOverflow
3/19/2018

Version 6.2.0 is supporting kubernetes auto discovery

update your elasticsearch.yml as following

discovery.zen.ping.unicast.hosts: "kubernetes service name"

-- krish
Source: StackOverflow
9/28/2015

Place the containers into a Kubernetes Service. The Kubernetes API makes an 'endpoints' API available that lists the IP addresses of all of the members of a service. This endpoint set will dynamically shrink and grow as you scale the number of pods.

You can access endpoints with:

kubectl get endpoints <service-name>

or directly via the Kubernetes API, see:

https://github.com/kubernetes/kubernetes/blob/master/examples/cassandra/java/src/io/k8s/cassandra/KubernetesSeedProvider.java#L106

for an example of how this was done for Cassandra.

-- brendan
Source: StackOverflow