Which framework supports MapReduce with dockers?

2/1/2021

I need to run some dockers in map-reduce mode:

For example:

  • I have a list of lists of values, which I want to split to multiple dockers:
The list [[100, 120, 130], [155, 156], [180, 191, 197, 199]]

will be splitted to 3 different dockers, each docker will get different list value

The results of the 3 dockers will be used as input to new docker (reduce mode).

  • I read this paper:

https://medium.com/better-programming/running-a-mapreduce-word-count-application-in-docker-using-python-sdk-ed2f1e27d424 which suggest similar problem. The author wrote code (python using docker package) for map-reduce and implemented it.

Is there any framework which do this dockers map-reduce ? or

I need to implement it ?

-- Boom
docker
kubernetes
mapreduce
python

0 Answers