K
Q

Not able to connect private github repository in apache ariflow helm chart in Kubernetes

10/9/2020

I am new working with Airflow and Kubernetes. I am trying to use apache Airflow in Kubernetes.

To deploy it I used this chart: https://github.com/apache/airflow/tree/master/chart.

I want to upload my dags in my github repository so: 

gitSync:
    enabled: true
    # git repo clone url
    # ssh examples ssh://git@github.com/apache/airflow.git
    # git@github.com:apache/airflow.git
    # https example: https://github.com/apache/airflow.git
    repo: https://github.com/mygithubrepository.git
    branch: master
    rev: HEAD
    root: "/git"
    dest: "repo"
    depth: 1
    # the number of consecutive failures allowed before aborting
    maxFailures: 0
    # subpath within the repo where dags are located
    # should be "" if dags are at repo root
    subPath: ""

Then I see that to use a private github repository I have to create a secret as is specified in the value.yml file:

# if your repo needs a user name password
    # you can load them to a k8s secret like the one below
    #   ---
    #   apiVersion: v1
    #   kind: Secret
    #   metadata:
    #     name: git-credentials
    #   data:
    #     GIT_SYNC_USERNAME: <base64_encoded_git_username>
    #     GIT_SYNC_PASSWORD: <base64_encoded_git_password>
    # and specify the name of the secret below
    #credentialsSecret: git-credentials

I am creating the secret:

apiVersion: v1
data:
  GIT_SYNC_USERNAME: bXluYW1l
  GIT_SYNC_PASSWORD: bXl0b2tlbg==
kind: Secret
metadata:
  name: git-credentials
  namespace: default

Then I use the secret name in the value.yml file:

repo: https://github.com/mygithubrepository.git
    branch: master
    rev: HEAD
    root: "/git"
    dest: "repo"
    depth: 1
    # the number of consecutive failures allowed before aborting
    maxFailures: 0
    # subpath within the repo where dags are located
    # should be "" if dags are at repo root
    subPath: ""
    # if your repo needs a user name password
    # you can load them to a k8s secret like the one below
    #   ---
    #   apiVersion: v1
    #   kind: Secret
    #   metadata:
    #     name: git-credentials
    #   data:
    #     GIT_SYNC_USERNAME: <base64_encoded_git_username>
    #     GIT_SYNC_PASSWORD: <base64_encoded_git_password>
    # and specify the name of the secret below
    credentialsSecret: git-credentials

but seems not bee working.

-- J.C Guzman
airflow
kubernetes
kubernetes-helm

Similar Questions

Kafka NodePort service is unreachable outside of cluster
Exclude specific hosts from ssl redirect in Kubernetes Nginx Ingress
How to create kubeconfig for Kubernetes Airflow worker pod launching KubernetesPodOperator
Google Kubernetes Engine: restore service account
Kubernetes - config map templated variables

2 Answers

9/7/2021

I ran into this exact issue. I was just about to use ssh instead, but then just persisted. How I got https to work (using a PAT) is:

Fill in your repo URL

repo: "https://github.com/USERNAME/REPOSITORY.git"

Add the name of your secret

httpSecret: "my-airflow-secret"

Fill in this part too

httpSecretUsernameKey: "your-username-key"
httpSecretPasswordKey: "your-PAT-key"

Now, the key is, when creating your base64 encoded strings to put in your secret, use:

echo -n "your-user-name" | base64

The -n means echo doesn't append a newline. After that It worked correctly. I have many other applications that didn't mind base64 encoded strings without -n so I guess I just got a bit complacent. It took me a while to realise the issue though.

-- Coffee and Code
Source: StackOverflow
10/15/2020

I see that you are connecting to your github repo via https.

Try to use:

ssh://git@github.com/mygithubrepository.git

or simply

git@github.com/mygithubrepository.git

You can experience issues with connecting via https especially if you have two-factor authentication enabled on your github account. It's described more in detail in this article.

Also take a look at VonC's answer where he mentions:

As noted in Oliver's answer, an HTTPS URL would not use username/password if two-factor authentication (2FA) is activated.

In that case, the password should be a PAT (personal access token) as seen in "Using a token on the command line".

That applies only for HTTPS URLS, SSH is not affected by this limitation.

As well as at this one provided by rc0r.

But as I said, simply using ssh instead of https should resolve your problem easily.

-- mario
Source: StackOverflow