I have a Elastic search setup.
I need to deploy the same in kubernetes
Purpose : Automatically scale up the ES server with all its data to 3 instances if the CPU/RAM reaches 90% and scale it down to one instance with all its data
Using ReplicationController
scaling can be possible
I need , How to setup ES with all it's current data in kubernetes
Can anyone help me on this out?
You will want StatefulSet
s (tutorial) for part of that problem (they honor kubectl scale sts --replicas=$foo
, just like an rc
)
Having members with predictable hostnames (as a StatefulSet
will do) makes the discovery.zen.ping.hostnames
a ton easier to configure -- even if there is only one ES node actually alive at the time (that is: you can specify my-es-0.my-es.my-ns.svc.cluster.local,my-es-1.my-es.my-ns.svc.cluster.local,etc
in the zen
list, and the first ES node will whine, but the 2nd and upward will benefit from the actual discovery part)
scale it down to one instance with all its data
Good luck with that; you'd have to re-balance the shards of every index with each resizing operation, since ES wants to distribute the load out across all members of its cluster -- the exact opposite of your consolidation objective.
I'd bet you could make a custom Horizontal Pod Autoscaler implementation that waited until the rebalancing operations had completed before destroying the Pods, but I'm not aware right now of any out-of-the-box mechanism for doing that.