I am working in microservices architecture. Every service has few long-running tasks (data processing, report generation) which can take up to 1-2 hours. We are using Kafka for the queue. How to handle cases where pod restart or any deployment happens just before the completion of the task? The task will start again and will take that much time again. Is there any way to run these tasks independently to the application pod?
you can use the Kubernetes jobs to do these types of tasks so as soon as the task is done Kubernetes will auto-delete the pods also.
Jobs are also configurable and will be running standalone so if you will deploy the job again it fetches the data from Kafka and jobs new job will start.