I am using Google Cloud and I am doing RnD whether we can apply HPA (Horizontal Pod Auto scaling) on elasticsearch in Kubernetes.
I did elasticsearch set up on Kubernetes : https://github.com/elastic/helm-charts/tree/master/elasticsearch
But I found one of the post on forum they say elasticsearch HPA is hard
https://discuss.elastic.co/t/how-to-scale-up-and-down-nodes-automatically/224089/2
So is it possible to do HPA on elasticsearch or not ?
I don't think it would work well and you'd be at risk of losing data. HPA tends to respond to load changes on a scale of a minute or so, and can occasionally make large changes (scaling from 5 replicas to 2, for example). For Elasticsearch you need to scale in one node at a time, monitor the state of the cluster before you can proceed, and it could take a long time before you can move from one node to the next.
Say you're running Elasticsearch in a StatefulSet. Remember that each ES index is made up of shards; you will have some number of copies of the shards spread out across the nodes in the ES cluster. So also let's say your index has 10 shards with 2 replicas each.
Scaling out is easy. Increase the size of the StatefulSet as much as you want; configure each node to talk to es-0 for discovery. ES will see that the cluster has grown and start moving shards on to the new nodes on its own.
Scaling in is hard. You can only delete one node at a time. Once that node is shut down and the cluster realizes it's missing, then the shards that used to be on that node will be under-replicated. ES will create new replicas on the remaining nodes and copy the shard data there. You can observe this happening through things like the /_cat/shards API. The index status will be "yellow" as long as there are under-replicated shards, and then will switch back to "green" once the replication sequence finishes.
So say you currently have 8 nodes, and you want to scale down to 6. There's some possibility the only two replicas of a given shard will be on es-6 and es-7 so you can't turn those both off together; you have to turn off es-7 first, wait for replication to catch up, then turn off es-6. There's also some possibility that, when you turn off es-7, the new replicas will be created on the doomed node es-6.
You can also tell Elasticsearch to move shards off of a replica before removing it. This avoids the cluster going into the "yellow" state, but it's harder to monitor.
Finally, the re-replication can take a long time depending on how much data is actually in the cluster. (In a past cluster I maintained with a badly-designed index structure, it took multiple hours to shut down a node.) That's a much slower operation than HPA is prepared for.