Can Airflow Macros be passed to the GKEPod operator parameters?

7/3/2019

I am trying to figure out if there is a useful way in airflow to pass the execution date in a dag to the GKEPodOperator parameters.

This is the code I have:

with DAG(
    DAG_NAME,
    default_args=default_args,
) as dag:


    end_date = '{{ ds }}'
    start_date = '{{ (ds - macros.timedelta(days=1)).strftime("%Y-%m-%d") }}'

    playlog_to_gcs = GKEPodOperator(
        task_id=f"test",
        project_id=PROJECT,
        location=LOCATION,
        cluster_name=CLUSTER_NAME,
        namespace="default",
        image=f"gcr.io/{PROJECT}/test",
        name=f"{DAG_NAME}",
        get_logs=True,
        # I would like to be able to use macros with strings to pass to parameters in operators such as this
        env_vars={
            "TARGET_PATH": f"test/test-{end_date}.csv",
        },
    )

However, TARGET_PATH ends up resulting in test/test-{{ ds }}.csv

-- Racheallav
airflow
google-kubernetes-engine
jinja2

1 Answer

7/5/2019

Templates are evaluated by Jinja2 in all templated fields, which are described in the operator class. In your case, it happens here: https://github.com/apache/airflow/blob/2bdb053db618de7064b527e6e3ebe29f220d857b/airflow/contrib/operators/gcp_container_operator.py#L228

and here : https://github.com/apache/airflow/blob/2bdb053db618de7064b527e6e3ebe29f220d857b/airflow/contrib/operators/kubernetes_pod_operator.py#L102

env_vars is indeed a templated fields so it should work. I think you should try instead: "test/test-{{ end_date }}.csv"

-- Breathe
Source: StackOverflow