K
Q

Mounting ConfigMap to Airflow Worker Pod with KubernetesExecutor

4/3/2020

I have a scenario where I want to use Helm to deploy per environment Airflow configurations that use the KubernetesExecutor.

The issue I am running into is I need to the Scheduler mount per environment ConfigMaps in the Worker Pods when the KubernetesExecutor spawns them.

I have those ConfigMaps mounted fine in the Web and Scheduler pods, but I don't see a means to tell the KubernetesExecutor to mount specific ConfigMaps.

I don't see anywhere in the documentation how to do this, but I can't be the only one who wants their Docker Images to be environment agnostic and abstract everything away to the Helm level.

Any suggestions would be much appreciated :)

Thanks!

-- Jon Battista
airflow
configmap
kubernetes-helm
kubernetesexecutor

Similar Questions

Dynamic route to a specific StateFulSet POD by hostname
.Net Core 3.+ OAuth Authentication for a distributed scenario in kubernetes
Deploy at Kubernetes a Docker Image with Frontend + Backend application
Client sent an HTTP request to an HTTPS server

1 Answer

4/6/2020

I believe the easiest way to do this is by specifying the env_from_configmap_ref setting in your airflow.cfg as described here. If the configmaps are the same name in your different environments, this should be quite straight forward, e.g.

[kubernetes]
...
env_from_configmap_ref = <your-airflow-configmap>
...

If you have configmaps with different names in different environments (and can't make naming consistent), then you may need to specify the airflow.cfg in a configmap itself like in this example.

-- WillySchu
Source: StackOverflow