Im trying to apply a config file to create a POD from Cloud Compose using the BashOperator
First I tried using the PodOperator but it doesnt allow to pass a spec file, it just builds from the image.
I tried using the BashOperator since the worker already includes gcloud, kubectl, etc. https://cloud.google.com/composer/docs/how-to/using/writing-dags#bashoperator
But every time I run a command I get some error with the arguments or with gcloud sdk.
I tried running single kubectl commands and kubectl is indeed installed on the worker.
I tried setting the config for the cluster before but I got errors on that call too.
job = BashOperator(
task_id='start',
bash_command='kubectl apply -f
/home/airflow/gcs/dags/.../spec.yaml',
env=prodigy_env
)
I expect the pod to be created effectively with the config file but the actual output is
{bash_operator.py:124} INFO - unable to recognize - Path to file
*If you run a cat /path the file indeed exists
I forgot to answer, apparently passing env vars through the bash operator was messing up with the execution.
The path exists and I was able to run the script without any problems I just couldn't pass env variables that way.
I ended up passing them in the spec.yaml
You are having issues probably because although you have gcloud auth configured you still have to get kubectl credentials for the cluster:
gcloud container clusters get-credentials $CLUSTER_NAME --zone $GCP_ZONE --project $PROJECT_IDRun this command before kubectl apply -f /home/airflow/gcs/dags/.../spec.yaml
The content of DAGs bucket Buckets/<your bucket>/dags/ is copied to '/home/airflow/gcsfuse/dags/' on workers preserving the folder structure.
If your spec.yaml is stored in your DAGs bucket like this Buckets/<your bucket>/dags/<subfolder1>/spec.yaml then the file should be available under the following path /home/airflow/gcsfuse/dags/<subfolder1>/spec.yaml.
Just taking a look on the "/home/airflow/gcs/dags/.../spec.yaml'" path from the file system perspective - instead of "..." you probably should use ".." to refer to folder above "dags" folder.
--
On the other hand, there is an alternative to creating a Pod using explicitly invoking kubectl command - namely you could use KubernetesPodOperator - this web page describes how to use KubernetesPodOperator to create Pods