Airflow on k8s and Google Operators: creds verification

2/29/2020

I have Apache Airflow on k8s.

Earlier, when Airflow was running on my local server (not k8s) i didn't have troubles with oauth2 creds verification: when Google Operators (based on GoogleCloudHook) starts, my browser opens and redirects me to Google Auth page. It was one-time procedure.

With Airflow on k8s my tasks running on separate pods and there are troubles with this oauth2 creds verification, i cant "open browser" inside pod, and i dont want to do it every time when my task will be running.

Can I somehow disable this procedure or automatizate this?

Is there any solution?

-- den
airflow
google-cloud-storage
kubernetes

1 Answer

3/1/2020

In order to authenticate you shoukd firstly be using the correct operator and executor in Airflow. In your case this would be the Kubernetes Executor. When using this executor you need to set up secret/s for use with k8s. Refer to the documentation here Kubernetes Executor
Overview

-- Paddy Popeye
Source: StackOverflow