I have a function to add tasks to different queues based on the user request. After tasks queued, there are some workers that listening to the queue will start doing the task.
For example, a user added 200 tasks to queue "A". Workers that listening queue "A" start doing tasks.
I want to spin up some workers that listening on the given Queue when someone added tasks to the queue.
For example, a user added 200 tasks to queue "A". The system then creates 3 workers listening to queue "A" and workers start doing tasks.
So my question is: How to spin up some pods (workers) in Kubernetes programmatically? or Is there any better solutions for my use case?
Other than already mentioned Knative I would also suggest two other options (listed from the more fitting in my opinion) that would allow you to scale up/down Pods programmatically:
The Horizontal Pod Autoscaler automatically scales the number of pods in a replication controller, deployment, replica set or stateful set based on observed CPU utilization (or, with custom metrics support, on some other application-provided metrics).
Here you will find a general walkthrough showing you how to set it up.
If you decide to scale based on metrics other than CPU / memory than you will need to use Custom Metrics.
This tutorial provides an introduction to managing applications with StatefulSets. It demonstrates how to create, delete, scale, and update the Pods of StatefulSets.
With it you can easily scale the number of replicas.
Please let me know if that helped.
If the worker pods are doing ephemeral short lived tasks then have a look at Knative. Knative is a serverless platform built on top of kubernetes which spins up pods bases on events coming from different sources.
Knative is implemented as a set of controllers you install on your Kubernetes cluster. Knative registers its own API types to the Kubernetes API, so working with it is not too different from working with Kubernetes itself.
Knative consists of two independent components that have their own GitHub projects:
Knative Serving: Run stateless services more easily on Kubernetes, by making autoscaling, networking and rollouts easier.
Knative Eventing: Create subscriptions to event sources declaratively, and route events to Kubernetes endpoints.
You can use either one or both components in your cluster.
Knative eventing is a system that is designed to address a common need for cloud native development and provides composable primitives to enable late-binding event sources and event consumers.