I have a spring batch application which will trigger the job to transfer bulk data from one database to another database through API call.All jobs are configured to work in parallel processing(Master/slave step)partition and deployed this application in openshift. Need to autoscale the application based on the load during the job execution.Even though i have used the openshift autoscale feature still i couldn't find the efficiency in performance of the job. PODs are simply creating but only one POD is getting utilized.How to fix this issue?How to split jobs among PODS?
In a remote partitioning setup, the master step sends StepExecutionRequests to a configurable queue (let's call it requests). Worker steps are listeners on this queue. The master step can be configured to either:
replies)With this in mind, autoscaling such a setup depends on how you define your PODs. For example, if you run one or multiple workers in the same POD, you can autoscale this deployment as the size of the requests queue grows.
Hope this helps.