Use kubernetes secret with GKEPodOperator in Airflow

12/16/2019

I am trying to use a GOOGLE_APPLICATION_CREDENTIALS secret with GKEPodOperator. Basically I want to: 1. Upload the secret to GKE 2. Mount (?) the secret to a container 3. Use the secret when running the container.

Until now I have added the key.json-file to my image at build time, and I know this is not the correct way to do it.

I found this question: How to set GOOGLE_APPLICATION_CREDENTIALS on GKE running through Kubernetes

The difference is that they are not using GKEPodOperator.

What I have done: 1. Created the secret using:

kubectl create secret generic mysupersecret --from-file=service_account_key=key.json

I see there are volumes and volume_mounts parameters but I dont understand how to use them.

Can anyone give me a helping hand on this? Maybe I am about to do something stupid..

-- wital
airflow
docker-secrets
google-kubernetes-engine
kubernetes
kubernetes-secrets

2 Answers

12/16/2019

To use a Secret with your workloads, you can specify environment variables that reference the Secret's values, or mount a volume containing the Secret. Please follow this link to using secrets and set volumes and volume_mounts.

This link refer to the Google general document for Authenticating to Cloud Platform with Service Accounts to use a GOOGLE_APPLICATION_CREDENTIALS secret. And this link describes how to use the KubernetesPodOperator to launch Kubernetes pods.

-- Ahmad P
Source: StackOverflow

2/21/2020

This is similar to passing secrets to the KubernetesPodOperator. Check details here.

Here is quick sample.

influx_username = secret.Secret(
...
)
influx_pass = secret.Secret(
...
)
operator = GKEPodOperator(
    task_id='task-id',
    project_id='prj-id',
    location='location',
    cluster_name='cluster-name',
    name='pod-name',
    namespace='default',
    image='image-path',
    secrets=[influx_username, influx_pass],
)
-- Mikhail Dudin
Source: StackOverflow