Anyone here have experience about batch processing (e.g. spring batch) on kubernetes ? Is it good idea ?

For Spring Batch, we (the Spring Batch team) do have some experience on the matter which we share in the following talks:

• Cloud Native Batch Processing on Kubernetes, by Michael Minella
• Spring Batch on Kubernetes, by me.

Running batch jobs on kubernetes can be tricky:

• pods may be re-scheduled by k8s on different nodes in the middle of processing
• cron jobs might be triggered twice
• etc

This requires additional non-trivial work on the developer's side to make sure the batch application is fault-tolerant (resilient to node failure, pod re-scheduling, etc) and safe against duplicate job execution in a clustered environment.

Spring Batch takes care of this additional work for you and can be a good choice to run batch workloads on k8s for several reasons:

• Cost efficiency: Spring Batch jobs maintain their state in an external database, which makes it possible to restart them from the last save point in case of job/node failure or pod re-scheduling
• Robustness: Safe against duplicate job executions thanks to a centralized job repository
• Fault-tolerance: Retry/Skip failed items in case of transient errors like a call to a web service that might be temporarily down or being re-scheduled in a cloud environment

How to prevent batch processing process same data if we use kubernetes auto scaling feature ?

Making each job process a different data set is the way to go (a job per file for example). But there are different patterns that you might be interested in, see Job Patterns from k8s docs.