GitLab pod. Readiness probe failed: Get http://10.233.124.46:8005/health_check

6/1/2018

Use https://docs.gitlab.com/ce/install/kubernetes/gitlab_omnibus.html#configuring-and-installing-gitlab for install GitLab. But I have a problem.

I see in describe gitlab pod: Warning Unhealthy 1m (x195 over 39m) kubelet, k8s-m3.me Readiness probe failed: Get http://10.233.105.52:8005/health_check?token=SXBAQichEJasbtDSygrD: dial tcp 10.233.105.52:8005: getsockopt: connection refused

Kubernetes v1.10.2

# kubectl get pods -owide
NAME                                        READY     STATUS             RESTARTS   AGE       IP              NODE
gitlab1-gitlab-75576c4589-7dvlm             0/1       CrashLoopBackOff   92         9h        10.233.105.52   k8s-m2.me
gitlab1-gitlab-postgresql-f66555d65-dk5v6   1/1       Running            0          9h        10.233.78.175   k8s-m1.me
gitlab1-gitlab-redis-58cf598657-mqs2w       1/1       Running            0          9h        10.233.124.48   k8s-m3.me
gitlab1-gitlab-runner-555b89f44b-7vgwm      0/1       CrashLoopBackOff   100        9h        10.233.105.51   k8s-m2.me

I see problem in next log (full log https://gist.github.com/IvanBiv/9a89024f5cd4a1207070badd025bf57c):

# kubectl describe pod gitlab1-gitlab-75576c4589-7dvlm
...
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason     Age                 From                Message
  ----     ------     ----                ----                -------
  Normal   Pulled     58m (x84 over 9h)   kubelet, k8s-m2.me  Container image "gitlab/gitlab-ce:10.6.2-ce.0" already present on machine
  Warning  Unhealthy  9m (x1709 over 9h)  kubelet, k8s-m2.me  Readiness probe failed: Get http://10.233.105.52:8005/health_check?token=SXBAQichEJasbtDSygrD: dial tcp 10.233.105.52:8005: getsockopt: connection refused
  Warning  BackOff    4m (x1064 over 8h)  kubelet, k8s-m2.me  Back-off restarting failed container

Gitlab Runner log

# kubectl logs gitlab1-gitlab-runner-555b89f44b-7vgwm
+ cp /scripts/config.toml /etc/gitlab-runner/
+ /entrypoint register --non-interactive --executor kubernetes
Running in system-mode.

ERROR: Registering runner... failed                 runner=tQtCbx5U status=couldn't execute POST against http://gitlab1-gitlab.default:8005/api/v4/runners: Post http://gitlab1-gitlab.default:8005/api/v4/runners: dial tcp 10.233.49.21:8005: i/o timeout
PANIC: Failed to register this runner. Perhaps you are having network problems

Services

# kubectl get services -owide
NAME                                                  TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                      AGE       SELECTOR
gitlab1-gitlab                                        ClusterIP   10.233.49.21    <none>        22/TCP,8065/TCP,8105/TCP,8005/TCP,9090/TCP   9h        name=gitlab1-gitlab
gitlab1-gitlab-postgresql                             ClusterIP   10.233.60.177   <none>        5432/TCP                                     9h        name=gitlab1-gitlab-postgresql
gitlab1-gitlab-redis                                  ClusterIP   10.233.43.12    <none>        6379/TCP                                     9h        name=gitlab1-gitlab-redis
-- Ivan
gitlab
kubernetes
kubernetes-helm

1 Answer

6/2/2018

Couple of times we had this issue. Instead of health check, we started to use login page where it sends 200. Use the context path below

  • /users/sign_in
-- Pamir Erdem
Source: StackOverflow