I have a gitlab runner build by helm on GKE, I had registration this runner.
When I trigger my pipelines, runner run failed and got this error
Running with gitlab-runner 11.7.0 (8bb608ff)
on gitlab-runner-gitlab-runner-5bb7b68b87-wsbzf -xsPNg33
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image docker ...
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Waiting for pod gitlab/runner--xspng33-project-3-concurrent-0rsbpp to be running, status is Pending
Running on runner--xspng33-project-3-concurrent-0rsbpp via gitlab-runner-gitlab-runner-5bb7b68b87-wsbzf...
Cloning into '/general/year-end-party/yep-web'...
Cloning repository...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.shopee.nctu.me/general/year-end-party/yep-web.git/': SSL certificate problem: unable to get issuer certificate
/bin/bash: line 72: cd: /general/year-end-party/yep-web: No such file or directory
ERROR: Job failed: command terminated with exit code 1
I saw many solutions say I could set ssl_verify false. But my runner is installed by helm, I didn't touch runner's config.toml. I don't know how could I solve this. Please help me.
I was having the same issue and in my case it turn out that I constructed the certificate chain wrong. These are the steps to verify that that you are using the right certificate file.
First you create a concatenated PEM file with the complete certificate chain (server certificate, intermediate CAs and root CA). The best approach is to have one separate file for each certificate so that you can verify that the chain in correct with openssl verify
before concatenation:
openssl verify -CAfile USERTrustRSACA.crt -untrusted SectigoRSAOrganizationValidationSecureServerCA.crt server.crt
server.crt: OK
If you have any error message like error 20 at 1 depth lookup:unable to get local issuer certificate
means that you don't have the right set of certificates. This can happen to you if you tried to download the CA certificates yourself using the CA Issuers - URI:http://crt.usertrust.com/USERTrustRSAAddTrustCA.crt
field in the certificate like I did.
Once you are able to validate the certificate chain locally with openssl
you can concatenate everything in a single file
cat server.crt SectigoRSAOrganizationValidationSecureServerCA.crt USERTrustRSACA.crt > gitlab.my.domain.com.crt
The file must match the FQDN of your server.
Now you can install this certificate on your GitLab server:
gitlab.my.domain.com.crt
to you GitLab server /etc/gitlab/ssl
(The certificate private key must also be there)sudo gitlab-ctl reconfigure
sudo gitlab-ctl hup nginx
Then verify that the change is in place with
openssl s_client -connect gitlab.my.domain.com:443 # look for "Certificate chain"
Then you can follow the step in Providing a custom certificate for accessing GitLab:
kubectl -n gitlabrunner delete secret gitlab-my-domain-com-cert
kubectl -n gitlabrunner create secret generic gitlab-my-domain-com-cert --from-file=gitlab.my.domain.com.crt
helm install --namespace gitlabrunner --set gitlabUrl=https://gitlab.my.domain.com --set runnerRegistrationToken=xxxxxxxxx --set logLevel=debug --set certsSecretName=gitlab-my-domain-com-cert gitlab-runner gitlab/gitlab-runner
The provided ca certificate is never used due to the unfixed bug #3968.
The workaround solution for this, is to explicityl reference your ca.crt
and config.toml
userspace paths of the runner container in the values.yaml
that you use for executing the HELM chart:
envVars:
- name: CI_SERVER_TLS_CA_FILE
value: /home/gitlab-runner/.gitlab-runner/certs/ca.crt
- name: CONFIG_FILE
value: /home/gitlab-runner/.gitlab-runner/config.toml
You need to create a Kubernetes Secret with the content of your certificate in the namespace of your gitlab-runner. The secret will be used to populate the /etc/gitlab-runner/certs directory in the gitlab-runner.
After that, you need to refer the secret name in your values.yaml helm chart :
## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /etc/gitlab-runner/certs directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates
##
certsSecretName: <name_of_your_secret>
More info in the gitlab documentation.