Elastic Search setup using Kubernetes

3/23/2018

I have a Elastic search setup.

I need to deploy the same in kubernetes

Purpose : Automatically scale up the ES server with all its data to 3 instances if the CPU/RAM reaches 90% and scale it down to one instance with all its data

Using ReplicationController scaling can be possible

I need , How to setup ES with all it's current data in kubernetes

Can anyone help me on this out?

-- soundararajan.c
docker
elasticsearch
kubernetes

1 Answer

3/24/2018

You will want StatefulSets (tutorial) for part of that problem (they honor kubectl scale sts --replicas=$foo, just like an rc)

Having members with predictable hostnames (as a StatefulSet will do) makes the discovery.zen.ping.hostnames a ton easier to configure -- even if there is only one ES node actually alive at the time (that is: you can specify my-es-0.my-es.my-ns.svc.cluster.local,my-es-1.my-es.my-ns.svc.cluster.local,etc in the zen list, and the first ES node will whine, but the 2nd and upward will benefit from the actual discovery part)

scale it down to one instance with all its data

Good luck with that; you'd have to re-balance the shards of every index with each resizing operation, since ES wants to distribute the load out across all members of its cluster -- the exact opposite of your consolidation objective.

I'd bet you could make a custom Horizontal Pod Autoscaler implementation that waited until the rebalancing operations had completed before destroying the Pods, but I'm not aware right now of any out-of-the-box mechanism for doing that.

-- mdaniel
Source: StackOverflow