I am using Airflow2.0 with KubernetesPodOperator want to run a command that use as a parameter a file from inside the image ran by the Operator. This is what I used:
KubernetesPodOperator(
namespace=commons.kubernetes_namespace,
labels=commons.labels,
image=f"myregistry.io/myimage:{config['IMAGE_TAG']}",
arguments=[
"python",
"run_module.py ",
"-i",
f'args/{config["INPUT_DIR"]}/{task_id}.json'
],
name=dag_name + task_id,
task_id=task_id,
secrets=[secret_volume]
)
But this gives me the error:
raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: args/airflow2test/processing-pipeline.jsonThe image does not use any macros.
Anyone has any clue? What do I do wrong?
This was a bug that started with PR released in version 2.0.0 of apache-airflow-providers-cncf-kubernetes.
The goal of the change was to allow templating of .json files. There was a GitHub issue about the problems it created. The bug was eventually resolved by PR which was released in version 2.0.2 of the provider.
Solution:
apache-airflow-providers-cncf-kubernetes (currently 2.0.2)KubernetesPodOperatorThere are two ways to workaround that problem one is to change template_fields the other is to change template_ext:
1st option: As posted on issue by raphaelauv is not to allow rendering of arguments field:
class MyKubernetesPodOperator(KubernetesPodOperator):
template_fields = tuple(x for x in KubernetesPodOperator.template_fields if x != "arguments")2st option: if you prefer not to render .json files:
class MyKubernetesPodOperator(KubernetesPodOperator):
template_ext = ('.yaml', '.yml',)