I have a web front end processing webhooks and sending messages to a backend worker. These messages belong to different organizations, each message has the following format:

class Message{
   int OrganizationId {get;set;}
   string Payload {get;set;}
} 

Today, the processing backend is implemented as an Azure Webjob. The webjob is simple console application intercepting the messages sent from the front end. The console application lists all organizations pending in the queue and creates a thread for each one so the messages can be processed in parallel for each organization.

Azure Webjobs do unfortunately not scale that well (the reasons are along the same lines covered in this article). We are considering to move the background processing tasks to Kubernetes. Ideally, each organization messages should be processed in a container. These containers will be grouped in groups of 5 in each pod.

While the pod configuration is simple, we can't figure out a way to automatically and continuously discover organizations in the message queue and fire off a container with the organization ID as a parameter.

Note: The current implementation does not allow us to split each organization messages into a separate queue as there can be thousands of organizations pumping messages to the queue at the same time.