I have a web server and celery beat worker that I am running on kubernetes. In deployment, I call this command -
helm upgrade X ./X --set xImageTag=$TAG,nginxImageTag=$TAG --install
The pods are being restarted. If there is a running celery task - it is stopped in the middle and lost forever. Is there a graceful way to prevent that from happening?
I am thinking about: using the pod PreStop hook OR creating something that will prevent the task from stopping.
I want the task to continue running even when deploying new code and calling
helm upgrade ..... --install.
I'm sure someone already had that problem, but I couldn't find anything online. Thanks!
We solved it by having the celery task programatically create a k8s job. Now the process of the celery task is executed in a k8s job, which won't be deleted in helm install. The celery task will create a k8s job, and die. The code for creating a k8s job programatically is here: blog.pythian.com/how-to-create-kubernetes-jobs-with-python .