I am using Kubernetes executor in airflow and for data management (sharing data across pod), trying to write data in some mounted directory but the issue that I am facing is how to mount PVC over the worker pod where that task executes.
Worker pod spawns only at the time of dag execution and get deleted after completion of the task. Since the worker pod is only manageable through the configuration file (airflow.cfg).
How can bind a PVC to worker pod other than used for dags and logs?
When you delete a PVC, corresponding PV becomes Released. This PV can contain sensitive data (say credit card numbers) and therefore nobody can ever bind to it, even if it is a PVC with the same name and in the same namespace as the previous one - who knows who's trying to steal the data!
Action is required here. You have two options:
Simply create new PVC bound to exiting PV (via its UID) and deployment that uses this PV.
Similar problem: pvc.
Official documentation: kubernetes-pvc.