I want to create a deployment kubernetes instead of a pod. Is there a way to do it in apache airflow? . I know about the KubernetesPodOperator, but I want the kind : Deployment and not as kind: Pod.
KubernetesPodOperator in Apache Airflow
k = KubernetesPodOperator(namespace='default',
image="ubuntu:16.04",
cmds=["bash", "-cx"],
arguments=["echo", "10"],
labels={"foo": "bar"},
secrets=[secret_file,secret_env]
volume=[volume],
volume_mounts=[volume_mount]
name="test",
task_id="task",
affinity=affinity,
is_delete_operator_pod=True,
hostnetwork=False,
tolerations=tolerations
)
Thanks
I want to create a deployment kubernetes instead of a pod. Is there a way to do it in apache airflow?
Not really. Apache Airflow is a tool to run batch tasks defined in a DAG which does not fit the pattern of running the workload in a Kubernetes Deployment. A Deployment is meant for long-running services type of applications that don't terminate unless there's a crash or you explicitly terminate/delete it (aka, generally stateless microservices)
The controller in Kubernetes that fits more with what Apache Airflow does is the Job controller which also creates a Pod when it runs. But that's where Airflow provides a richer feature set of features like support for DAGs
In summary, having the ability to create a Deployment using Airflow is more a nice to have, but doesn't necessarily fit into the tool's main pattern usage. I suggest using a different tool/method to create your deployments.