We are using Kubernetes and we need to do "Smart partitioning" of data. We want to split the space between 1 to 1000 between n running Pods, And each pod should know which part of the space is his to handle (for pooling partitioned tasks).
So for example, if we have 1 pod he will handle the whole space from 1-1000.
When we scale out to 3 pods, each of them will get the same share.
Pod 1 - will handle 1-333
Pod 2 - 334-667
Pod 3 667-1000
Right now the best way that we find to handle this issue is to create a Stateful-set, that pooling the number of running pods and his instance number and decide which part of the space he needs to handle. Is there a smarter/built-in way in Kubernetes to partition the space between nodes in this manner?
Service fabric has this feature built-in.
There are NO native tools for scaling at the partition level in K8s yet.
Only custom solutions similar to what you have came up with in your original post.