K
Q

Gitlab CI Builds failing with 502 error in Gitlab Helm Deployment

11/2/2019

I recently set up GitLab using Helm in an on-prem kubernetes cluster. It works fine. Can access all aspects of the web ui, can SSH into it via external ingress controller (deployed separately from gitlab) just fine.

But when I try to run a job, I get the following error.

Running with gitlab-runner 12.4.1 (05161b14)
on gitlab-gitlab-runner-6db97976bb-bsfqj SFKvKAyD
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image node:6 ...
Waiting for pod gitlab/runner-sfkvkayd-project-1-concurrent-09tsz7 to be running, status is Pending
Waiting for pod gitlab/runner-sfkvkayd-project-1-concurrent-09tsz7 to be running, status is Pending
Running on runner-sfkvkayd-project-1-concurrent-09tsz7 via gitlab-gitlab-runner-6db97976bb-bsfqj...
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/redacted/web/.git/
Created fresh repository.
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.example.com/redacted/web.git/': The requested URL returned error: 502
ERROR: Job failed: command terminated with exit code 1

Why would I be getting a 502 error?

nginx-ingress values.yml:

controller:
  config:
    resolver-address: 10.0.0.1
    hsts-include-subdomains: "false"
    server-name-hash-bucket-size: "256"
    enable-vts-status: "true"
    use-http2: "false"
    ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
    ssl-protocols: "TLSv1.1 TLSv1.2"
    server-tokens: "false"

tcp:
  22: "gitlab/gitlab-gitlab-shell:22"

Gitlab values.yml:

global:
  edition: ee
  hosts:
    domain: example.com
    https: true
    gitlab:
      name: gitlab.example.com
      https: true
    minio:
      name: minio.example.com
      https: false    
  ingress:
    configureCertmanager: false
    enabled: true
    tls:
      enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
      kubernetes.io/tls-acme: true
  gitaly:
    persistence:
      size: 4Gi
  minio:
    enabled: true
  grafana:
    enabled: false
  appConfig:
    ldap:
      servers:
        main:
          label: 'LDAP'
          host: 'ipa.example.com'
          port: 389
          uid: 'uid'
          base: 'dc=example,dc=com'
          bind_dn: 'uid=system,cn=sysaccounts,cn=etc,dc=example,dc=com'
          password:
            secret: ldap-bind-secret
            key: ldap-password
          encryption: 'plain'
  registry:
    enabled: false
    bucket: registry

gitlab:
  unicorn:
    ingress:
      tls:
        secretName: gitlab-unicorn-tls

upgradeCheck:
  enabled: false    
certmanager:
  install: false
nginx-ingress:
  enabled: false
prometheus:
  install: false
redis:
  persistence:
    size: 1Gi
postgresql:
  install: true
  persistence:
    size: 1Gi
registry:
  enabled: false
gitlab-runner:
  install: true
  rbac:
    create: true
  runners:
    locked: false
    cache:
      cacheType: s3
      s3BucketName: runner-cache
      cacheShared: true
      s3BucketLocation: us-east-1
      s3CachePath: gitlab-runner
      s3CacheInsecure: false
minio:
  persistence:
    size: 4Gi
gitaly:
  persistence:
    size: 4Gi

Edit:

Log from the nginx-ingress-controller:

10.0.10.1 - - [02/Nov/2019:06:49:45 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 722 0.004 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 0 0.004 204 7c34fc2038325a786d949c9fdc82915b

(line is repeated several times)

Edit 2: Full logs from the moment I hit 'retry' on the job.

10.0.10.1 - - [04/Nov/2019:01:05:04 +0000] "POST /redacted/web/-/jobs/7/retry HTTP/1.1" 302 123 "https://gitlab.example.com/redacted/web/-/jobs/7" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1279 0.492 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 123 0.492 302 580cf6d7036706ea9f9182a5ed2385d6
10.0.10.1 - - [04/Nov/2019:01:05:05 +0000] "GET /redacted/web/-/jobs/8 HTTP/1.1" 200 9053 "https://gitlab.example.com/redacted/web/-/jobs/7" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1020 0.593 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 35481 0.596 200 5fb58ca0be4512efd80cea567bbd3127
10.0.10.1 - - [04/Nov/2019:01:05:05 +0000] "GET /redacted/web/-/jobs/8/trace.json?state= HTTP/1.1" 200 139 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1011 0.171 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 139 0.168 200 1b5673807d48539dad6765a9b88970e0
10.0.10.1 - - [04/Nov/2019:01:05:06 +0000] "GET /redacted/web/-/jobs/8.json HTTP/1.1" 200 1475 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 998 1.154 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 4158 1.156 200 17a2c976b8fc2e93f252be68281aadc6
10.0.10.1 - - [04/Nov/2019:01:05:07 +0000] "GET /redacted/web/pipelines/1/stage.json?stage=build&retried=1 HTTP/1.1" 200 1233 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1082 0.591 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 8059 0.593 200 4ba12adb7acf8635b40aac0d6c761797
10.0.10.1 - - [04/Nov/2019:01:05:10 +0000] "GET /redacted/web/-/jobs/8/trace.json?state= HTTP/1.1" 304 0 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1064 0.174 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 0 0.172 304 d7f47ceeaf4e501c7b97e7c5a65f5ad0
10.0.10.1 - - [04/Nov/2019:01:05:14 +0000] "GET /redacted/web/-/jobs/8/trace.json?state= HTTP/1.1" 304 0 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1064 0.531 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 0 0.528 304 7cccd15f2a3b5d57e22245b20b455962
10.0.10.1 - - [04/Nov/2019:01:05:17 +0000] "GET /redacted/web/-/jobs/8.json HTTP/1.1" 304 0 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1051 0.689 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 0 0.692 304 144e9352bed9f000e78e9336c159761c
10.0.10.1 - - [04/Nov/2019:01:05:18 +0000] "GET /redacted/web/-/jobs/8/trace.json?state= HTTP/1.1" 200 139 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1064 0.250 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 139 0.248 200 ec6b33ca557c52a0547d1ec4c3b6b564
10.0.10.1 - - [04/Nov/2019:01:05:19 +0000] "POST /api/v4/jobs/request HTTP/1.1" 201 6432 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 722 0.540 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 6432 0.540 201 08addd7e961ddd2c124c938ed4ec8d01
10.0.10.1 - - [04/Nov/2019:01:05:19 +0000] "POST /api/v4/jobs/request HTTP/1.1" 204 0 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 722 0.130 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 0 0.132 204 745513af8de06310ff8efba16b99a795
10.0.10.1 - - [04/Nov/2019:01:05:22 +0000] "PATCH /api/v4/jobs/8/trace HTTP/1.1" 202 7 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 721 0.098 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 7 0.100 202 cf049526b5190c8d540b67bcf2d1a4b3
10.0.10.1 - - [04/Nov/2019:01:05:23 +0000] "GET /redacted/web/-/jobs/8/trace.json?state= HTTP/1.1" 200 645 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1064 0.219 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 1501 0.216 200 64a0d629bd31b4041349a4dd389a043b
[04/Nov/2019:01:05:24 +0000]TCP200390120.026
10.0.10.1 - - [04/Nov/2019:01:05:25 +0000] "PATCH /api/v4/jobs/8/trace HTTP/1.1" 202 7 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 373 0.080 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 7 0.080 202 6bd14f3c6e7f7be2b638b5c1199beaa0
10.0.10.1 - - [04/Nov/2019:01:05:27 +0000] "PATCH /api/v4/jobs/8/trace HTTP/1.1" 202 8 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 976 0.176 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 8 0.176 202 1c6327ae37886df62f7ce1bd824d2e43
10.0.10.1 - - [04/Nov/2019:01:05:27 +0000] "GET /redacted/web/-/jobs/8/trace.json?state=eyJvZmZzZXQiOjQ1MCwibl9vcGVuX3RhZ3MiOjAsImZnX2NvbG9yIjpudWxsLCJiZ19jb2xvciI6bnVsbCwic3R5bGVfbWFzayI6MCwic2VjdGlvbnMiOlsicHJlcGFyZS1zY3JpcHQiXSwibGluZW5vX2luX3NlY3Rpb24iOjF9 HTTP/1.1" 200 868 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1183 0.233 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 2223 0.232 200 98a2b255a8854c49d10769132f6889c8
10.0.10.1 - - [04/Nov/2019:01:05:27 +0000] "PUT /api/v4/jobs/8 HTTP/1.1" 200 4 "-" "gitlab-runner 12.4.1 (12-4-stable; go1.10.8; linux/amd64)" 691 0.267 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 4 0.268 200 0ed6eb1dce657e0b0c711dede3c4bcd9
10.0.10.1 - - [04/Nov/2019:01:05:28 +0000] "GET /redacted/web/-/jobs/8.json HTTP/1.1" 200 1502 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1051 0.695 [gitlab-gitlab-unicorn-8181] [] 10.42.0.179:8181 4444 0.696 200 75058bd730898e877034a10a7e386300
10.0.10.1 - - [04/Nov/2019:01:05:32 +0000] "GET /redacted/web/-/jobs/8/trace.json?state=eyJvZmZzZXQiOjEyNTIsIm5fb3Blbl90YWdzIjowLCJmZ19jb2xvciI6bnVsbCwiYmdfY29sb3IiOm51bGwsInN0eWxlX21hc2siOjAsInNlY3Rpb25zIjpbXSwibGluZW5vX2luX3NlY3Rpb24iOjF9 HTTP/1.1" 200 264 "https://gitlab.example.com/redacted/web/-/jobs/8" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36" 1163 1.050 [gitlab-gitlab-unicorn-8181] [] 10.42.0.184:8181 283 1.048 200 706cf9b018952ffd2d875a4a060edb77
-- cclloyd
continuous-integration
gitlab
kubernetes
kubernetes-helm
nginx

Similar Questions

unable to recognize "calico.yaml": no matches for kind "Deployment" in version "apps/v1beta1"
Kubernetes and Kafka (Advantages, Disadvantages, and main differences)
Where are the Kubernetes kubelet logs located for the AWS EKS Clusters?
Getting notification on file modification within a container in kubernetes pod

0 Answers