I am trying to figure out if there is a useful way in airflow to pass the execution date in a dag to the GKEPodOperator parameters.
This is the code I have:
with DAG(
DAG_NAME,
default_args=default_args,
) as dag:
end_date = '{{ ds }}'
start_date = '{{ (ds - macros.timedelta(days=1)).strftime("%Y-%m-%d") }}'
playlog_to_gcs = GKEPodOperator(
task_id=f"test",
project_id=PROJECT,
location=LOCATION,
cluster_name=CLUSTER_NAME,
namespace="default",
image=f"gcr.io/{PROJECT}/test",
name=f"{DAG_NAME}",
get_logs=True,
# I would like to be able to use macros with strings to pass to parameters in operators such as this
env_vars={
"TARGET_PATH": f"test/test-{end_date}.csv",
},
)
However, TARGET_PATH ends up resulting in test/test-{{ ds }}.csv
Templates are evaluated by Jinja2 in all templated fields, which are described in the operator class. In your case, it happens here: https://github.com/apache/airflow/blob/2bdb053db618de7064b527e6e3ebe29f220d857b/airflow/contrib/operators/gcp_container_operator.py#L228
env_vars is indeed a templated fields so it should work. I think you should try instead: "test/test-{{ end_date }}.csv"