GitLab Runner cannot clone, unable to get issuer certificate

2/21/2019

I have a gitlab runner build by helm on GKE, I had registration this runner.

When I trigger my pipelines, runner run failed and got this error

Running with gitlab-runner 11.7.0 (8bb608ff)
  on gitlab-runner-gitlab-runner-5bb7b68b87-wsbzf -xsPNg33
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image docker ...
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Running on runner--xspng33-project-3-concurrent-0rsbpp via gitlab-runner-gitlab-runner-5bb7b68b87-wsbzf...
Cloning into '/general/year-end-party/yep-web'...
Cloning repository...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.shopee.nctu.me/general/year-end-party/yep-web.git/': SSL certificate problem: unable to get issuer certificate
/bin/bash: line 72: cd: /general/year-end-party/yep-web: No such file or directory
ERROR: Job failed: command terminated with exit code 1

I saw many solutions say I could set ssl_verify false. But my runner is installed by helm, I didn't touch runner's config.toml. I don't know how could I solve this. Please help me.

I also had add cert for runner enter image description here enter image description here

-- Rukeith
gitlab
gitlab-ci
gitlab-ci-runner
kubernetes
kubernetes-helm

3 Answers

5/13/2020

I was having the same issue and in my case it turn out that I constructed the certificate chain wrong. These are the steps to verify that that you are using the right certificate file.

First you create a concatenated PEM file with the complete certificate chain (server certificate, intermediate CAs and root CA). The best approach is to have one separate file for each certificate so that you can verify that the chain in correct with openssl verify before concatenation:

openssl verify -CAfile USERTrustRSACA.crt -untrusted SectigoRSAOrganizationValidationSecureServerCA.crt server.crt
server.crt: OK 

If you have any error message like error 20 at 1 depth lookup:unable to get local issuer certificate means that you don't have the right set of certificates. This can happen to you if you tried to download the CA certificates yourself using the CA Issuers - URI:http://crt.usertrust.com/USERTrustRSAAddTrustCA.crt field in the certificate like I did.

Once you are able to validate the certificate chain locally with openssl you can concatenate everything in a single file

cat server.crt SectigoRSAOrganizationValidationSecureServerCA.crt USERTrustRSACA.crt > gitlab.my.domain.com.crt

The file must match the FQDN of your server.

Now you can install this certificate on your GitLab server:

  • Copy the gitlab.my.domain.com.crt to you GitLab server /etc/gitlab/ssl (The certificate private key must also be there)
  • sudo gitlab-ctl reconfigure
  • sudo gitlab-ctl hup nginx

Then verify that the change is in place with

openssl s_client -connect gitlab.my.domain.com:443 # look for "Certificate chain"

Then you can follow the step in Providing a custom certificate for accessing GitLab:

kubectl -n gitlabrunner delete secret gitlab-my-domain-com-cert
kubectl -n gitlabrunner create secret generic gitlab-my-domain-com-cert --from-file=gitlab.my.domain.com.crt
helm install --namespace gitlabrunner --set gitlabUrl=https://gitlab.my.domain.com --set runnerRegistrationToken=xxxxxxxxx --set logLevel=debug --set certsSecretName=gitlab-my-domain-com-cert gitlab-runner gitlab/gitlab-runner
-- RubenLaguna
Source: StackOverflow

9/9/2019

The provided ca certificate is never used due to the unfixed bug #3968.

The workaround solution for this, is to explicityl reference your ca.crt and config.toml userspace paths of the runner container in the values.yaml that you use for executing the HELM chart:

envVars:
  - name: CI_SERVER_TLS_CA_FILE
    value: /home/gitlab-runner/.gitlab-runner/certs/ca.crt
  - name: CONFIG_FILE
    value: /home/gitlab-runner/.gitlab-runner/config.toml
-- Martin Peter
Source: StackOverflow

2/21/2019

You need to create a Kubernetes Secret with the content of your certificate in the namespace of your gitlab-runner. The secret will be used to populate the /etc/gitlab-runner/certs directory in the gitlab-runner.

After that, you need to refer the secret name in your values.yaml helm chart :

## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /etc/gitlab-runner/certs directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates
##
 certsSecretName: <name_of_your_secret>

More info in the gitlab documentation.

-- Nicolas Pepinster
Source: StackOverflow