I have a queue which filled (consistently) with work-item by one of my k8s pod.
I want to use k8s job to process each work item as I want each job to be handle by new pod as I want to use multiple container as suggest here, but jobs don't seem to support an infinite number of completions.
When I use spec.completions: null I got BackoffLimitExceeded.
Any Idea how to implement job without the need to specify the number of work item?
Is there alternative to job for implementing background worker in k8s ?
Thank you
My suggestion is to use Kubernetes resources in the way they have been designed: Job resources are one-off tasks that can ben triggered many times but they differ from the background jobs you're eager to implement.
If your application is popping jobs from a queue/backend, it's better to put in a Deployment with a for loop (YMMV according to programming language) and, eventually, scale it to down with another component if you don't want to allocate unused resources.
Another solution could be the specialization of each Job, using UUID as the name of the jobs and labelling them in order to group them: anyway, first suggestion of using Kubernetes in the Kubernetes way is strongly recommended.