initializing a new service when the number of jobs ina GCP pub/sub is too many

7/4/2018

I am working on a project on GCP and I need to create a system that works like a load balancer, but the load is the number of items in a pub/sub queue.

Here is more detail:

  1. I have a message queue which is based on pub/sub.
  2. A lot of messages are posted to this ques and I have one service which consume them.
  3. It would takes some hours to process each items in the queue.
  4. I want to start a new service ( a docker image) when the number of items in the queue became very big (say I want to start anew service when the number of items in the queue became more than 10 items and start another one when the number of items in the queue became more than 20 and so on) and shutdown the services when the number of items in the queue reduces (so for example if the number of items in the queue is come to under 20, shut down all services and only 2 services became live)

Now my questions are:

  1. How can I do this?
  2. Is kubernties a good solution? if yes, where can I find more information about it.
  3. Can I do it by pub/sub? If yes, where can I find information?
-- mans
google-cloud-platform
google-kubernetes-engine
kubernetes

1 Answer

7/4/2018

I have a similar requirement (that I am struggling with). I would suggest you look at an horizontal pod autoscaler based on an external stackdriver monitoring metric to see if it will meet your needs. This process is discussed here:

https://cloudplatform.googleblog.com/2018/05/Beyond-CPU-horizontal-pod-autoscaling-comes-to-Google-Kubernetes-Engine.html

and here:

https://cloud.google.com/kubernetes-engine/docs/tutorials/external-metrics-autoscaling

-- sirrus
Source: StackOverflow