I have an old algorithm that executes two steps, say A and B. A deals with connecting to an external service and retrieve data points. A needs to be executed n times, and all the data so gathered is passed as input to B. To help scale the A part, it was implemented using multi-threading where 10 threads are spawned and each connects to n/10 endpoints. Once all threads complete execution, the complete dataset is provided as input to B.

We are planning to create a Docker image of this algo. While we do that I would like to explore if we can do away with the multi threading and instead deploy multiple containers. This gives us better scalability as n is a variable and sometimes is very small or very large.

I could use Kubernetes to orchestrate these containers. However, I see two challenges: 1. How do I gather back all the data points into my core algo? 2. How do I know that all containers have finished processing, and that I could move to step B?

Any pointers or help is appreciated.