I'm using Google Composer however due to the fact that I'm trying to create a dynamically created DAG, I'm unable to use the managed web server [1]. So in the end I need to add a plugins because I need to access the Airflow Webserver using REST API, so I install the following plugins, because the experimental REST API is not sufficient for my case [2]:
My managed Airflow Webserver was able to pick up changes when I updated my plugins. However my self managed Airflow Webserver was unable to pick up the changes.
I understand that I need to restart my Airflow Webserver, however it seems like my pods are not got restarted after I perform this command:
kubectl get deploy airflow-webserver -o yaml | kubectl replace --force -f -For your reference my deployment.yaml can be seen if you're doing the steps mentioned in [3].
Restarting the pods by using kubectl patch for certain unused environment variable also seems to be not working.
[1] Dynamically Creating DAG based on Row available on DB Connection
[2] https://github.com/apache/airflow/blob/master/docs/api.rst
[3] https://cloud.google.com/composer/docs/how-to/managing/deploy-webserver
This issue is already solved by creating a new environment and then migrate all the data to new environment using [1], current version is 1.4.1 (the previous version that I use is 1.3.0). Information about release can be found in [2]. Plugins can be installed by copying the plugins directory to plugins directory using gcsfuse or using the web console.
If you're using plugins for your webserver it means your endpoint also protected by IAP. To access it programmatically, you can use service account [3].
Please note that when you're using a beta version. The one that created by using the following command:
gcloud beta composer environments create <env> --location=<location> --airflow-version <version> --node-count <node-count>You should first create the environments and then run the script I provided because this beta version is available on v1beta1. Meanwhile the script is only supporting v1 API.
Please also note that if you're using other k8s resources on your own inside the old environment, this won't be backed up by the script I mentioned. The script only back up all config to your airflow, ie: DB, DAG, and logs. So in case you're using a self manage webserver you should recreate your deployment for this webserver.