I'm setting up an Airflow environment on Google Cloud Composer for testing. I've added some secrets to my namespace, and they show up fine:
$ kubectl describe secrets/eric-env-vars
Name: eric-env-vars
Namespace: eric-dev
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
VERSION_NUMBER: 6 bytes
I've referenced this secret in my DAG definition file (leaving out some code for brevity):
env_var_secret = Secret(
deploy_type='env',
deploy_target='VERSION_NUMBER',
secret='eric-env-vars',
key='VERSION_NUMBER',
)
dag = DAG('env_test', schedule_interval=None, start_date=start_date)
operator = KubernetesPodOperator(
name='k8s-env-var-test',
task_id='k8s-env-var-test',
dag=dag,
image='ubuntu:16.04',
cmds=['bash', '-cx'],
arguments=['env'],
config_file=os.environ['KUBECONFIG'],
namespace='eric-dev',
secrets=[env_var_secret],
)
But when I run this DAG, the VERSION_NUMBER
env var isn't printed out. It doesn't look like it's being properly linked to the pod either (apologies for imprecise language, I am new to both Kubernetes and Airflow). This is from the Airflow task log of the pod creation response (also formatted for brevity/readability):
'env': [
{
'name': 'VERSION_NUMBER',
'value': None,
'value_from': {
'config_map_key_ref': None,
'field_ref': None,
'resource_field_ref': None,
'secret_key_ref': {
'key': 'VERSION_NUMBER',
'name': 'eric-env-vars',
'optional': None}
}
}
]
I'm assuming that we're somehow calling the constructor for the Secret
wrong, but I am not entirely sure. Guidance appreciated!
Turns out this was a misunderstanding of the logs!
When providing an environment variable to a Kubernetes pod via a Secret, that value
key in the API response is None
because the value comes from the secret_key_ref
.