Creating Deployment in Kubernetes with Apache Airflow

3/20/2019

I want to create a deployment kubernetes instead of a pod. Is there a way to do it in apache airflow? . I know about the KubernetesPodOperator, but I want the kind : Deployment and not as kind: Pod.

KubernetesPodOperator in Apache Airflow

k = KubernetesPodOperator(namespace='default',
                          image="ubuntu:16.04",
                          cmds=["bash", "-cx"],
                          arguments=["echo", "10"],
                          labels={"foo": "bar"},
                          secrets=[secret_file,secret_env]
                          volume=[volume],
                          volume_mounts=[volume_mount]
                          name="test",
                          task_id="task",
                          affinity=affinity,
                          is_delete_operator_pod=True,
                          hostnetwork=False,
                          tolerations=tolerations
                          )

Thanks

-- Deep Patel
airflow
deployment
kubernetes
pod

1 Answer

3/20/2019

I want to create a deployment kubernetes instead of a pod. Is there a way to do it in apache airflow?

Not really. Apache Airflow is a tool to run batch tasks defined in a DAG which does not fit the pattern of running the workload in a Kubernetes Deployment. A Deployment is meant for long-running services type of applications that don't terminate unless there's a crash or you explicitly terminate/delete it (aka, generally stateless microservices)

The controller in Kubernetes that fits more with what Apache Airflow does is the Job controller which also creates a Pod when it runs. But that's where Airflow provides a richer feature set of features like support for DAGs

In summary, having the ability to create a Deployment using Airflow is more a nice to have, but doesn't necessarily fit into the tool's main pattern usage. I suggest using a different tool/method to create your deployments.

-- Rico
Source: StackOverflow