K
Q

Question

GoCD in Kubernetes(minikube) in Windows 10, container error CrashLoopBackOff, exit code: 111

2/22/2019

I install GoCD using helm in Kubernetes and found that pod status is CrashLoopBackOff. I am using minikube with hyper-v in Windows 10 Pro. What's the problem? How can I find error root? Where can I find definition for exit code 111?

Here's my kubectl describe pod info:

Name:               gocd-app-server-7fd8f8b48d-r6sn9
Namespace:          gocd
Priority:           0
PriorityClassName:  <none>
Node:               minikube/192.168.88.82
Start Time:         Thu, 21 Feb 2019 15:57:15 +0700
Labels:             app=gocd
                    component=server
                    pod-template-hash=7fd8f8b48d
                    release=gocd-app
Annotations:        <none>
Status:             Running
IP:                 172.17.0.16
Controlled By:      ReplicaSet/gocd-app-server-7fd8f8b48d
Containers:
  gocd-server:
    Container ID:   docker://907271ebbe383b533ef1eb892021eaabc4cf6264a7052e2e453fcf97fdb28de7
    Image:          gocd/gocd-server:v19.1.0
    Image ID:       docker-pullable://gocd/gocd-server@sha256:34204533eb0e0c6f7544c6aa29f2da815d972bd22124b32307ca4ca8f40abd61
    Ports:          8153/TCP, 8154/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    111
      Started:      Fri, 22 Feb 2019 09:55:42 +0700
      Finished:     Fri, 22 Feb 2019 09:55:43 +0700
    Ready:          False
    Restart Count:  4
    Liveness:       http-get http://:8153/go/api/v1/health delay=90s timeout=1s period=15s #success=1 #failure=10
    Readiness:      http-get http://:8153/go/api/v1/health delay=90s timeout=1s period=15s #success=1 #failure=10
    Environment:
      GOCD_PLUGIN_INSTALL_kubernetes-elastic-agents:        https://github.com/gocd/kubernetes-elastic-agents/releases/download/2.1.0-123/kubernetes-elastic-agent-2.1.0-123.jar
      GOCD_PLUGIN_INSTALL_docker-registry-artifact-plugin:  https://github.com/gocd/docker-registry-artifact-plugin/releases/download/1.0.0-25/docker-registry-artifact-plugin-1.0.0-25.jar
    Mounts:
      /docker-entrypoint.d from goserver-vol (rw)
      /godata from goserver-vol (rw)
      /home/go from goserver-vol (rw)
      /preconfigure_server.sh from config-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from gocd-app-token-kbdjx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-vol:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      gocd-app
    Optional:  false
  goserver-vol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  gocd-app-server
    ReadOnly:   false
  gocd-app-token-kbdjx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  gocd-app-token-kbdjx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

I also checked nodes condition using kubectl describe nodes, here's the result

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 22 Feb 2019 10:25:48 +0700   Thu, 14 Feb 2019 18:16:55 +0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 22 Feb 2019 10:25:48 +0700   Thu, 14 Feb 2019 18:16:55 +0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 22 Feb 2019 10:25:48 +0700   Thu, 14 Feb 2019 18:16:55 +0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 22 Feb 2019 10:25:48 +0700   Thu, 21 Feb 2019 13:22:10 +0700   KubeletReady                 kubelet is posting ready status

Nodes condition seems to be alright.

Thank you for your assistance.

-- roscoe_x

go-cd

kubernetes

minikube

1 Answer

3/9/2019

Try to check preconfigure logs while pods still running:

kubectl -n gocd exec -it gocd-app-server-68bf7759b9-9dmr5 -c gocd-server cat  /godata/logs/preconfigure.log
kubectl -n gocd exec -it gocd-app-server-68bf7759b9-9dmr5 -c gocd-server cat  /godata/logs/preconfigure_complete.log

I had some error in end, different code but, it could helps to find your issue:

checking if server has already been configured
No configuration found in cruise-config.xml. Using default preconfigure_server scripts to configure server
Trying to create an elastic profile now.
HTTP/1.1 200 OK
Date: Sat, 09 Mar 2019 14:59:05 GMT
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-UA-Compatible: chrome=1
Set-Cookie: JSESSIONID=node01oiy3yzw700oc1kq79gazxg27m1.node0;Path=/go;Expires=Sat, 23-Mar-2019 14:59:05 GMT;Max-Age=1209600;HttpOnly
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: max-age=0, private, must-revalidate
Content-Type: application/vnd.go.cd.v1+json;charset=utf-8
ETag: "f710a37eb9543dcd53aa78be26c62ae6"
X-Runtime: 1248
Vary: Accept-Encoding, User-Agent
Transfer-Encoding: chunked

{
  "_links" : {
    "self" : {
      "href" : "http://localhost:8153/go/api/elastic/profiles/demo-app"
    },
    "doc" : {
      "href" : "https://api.gocd.org/19.2.0/#elastic-agent-profiles"
    },
    "find" : {
      "href" : "http://localhost:8153/go/api/elastic/profiles/:profile_id"
    }
  },
  "id" : "demo-app",
  "plugin_id" : "cd.go.contrib.elasticagent.kubernetes",
  "properties" : [ {
    "key" : "Image",
    "value" : "gocd/gocd-agent-docker-dind:v19.2.0"
  }, {
    "key" : "PodConfiguration",
    "value" : "apiVersion: v1\nkind: Pod\nmetadata:\n  name: pod-name-prefix-{{ POD_POSTFIX }}\n  labels:\n    app: web\nspec:\n  serviceAccountName: default\n  containers:\n    - name: gocd-agent-container-{{ CONTAINER_POSTFIX }}\n      image: gocd/gocd-agent-docker-dind:v19.2.0\n      securityContext:\n        privileged: true"
  }, {
    "key" : "SpecifiedUsingPodConfiguration",
    "value" : "true"
  }, {
    "key" : "Privileged",
    "value" : "true"
  } ]
}Trying to configure plugin settings.
HTTP/1.1 100 Continue

checking if server has already been configured
No configuration found in cruise-config.xml. Using default preconfigure_server scripts to configure server
Trying to create an elastic profile now.
checking if server has already been configured
No configuration found in cruise-config.xml. Using default preconfigure_server scripts to configure server
Trying to create an elastic profile now.
checking if server has already been configured
No configuration found in cruise-config.xml. Using default preconfigure_server scripts to configure server
Trying to create an elastic profile now.
/ # command terminated with exit code 137

also there some other logs files which could help you to define it:

/ # ls -l /godata/logs/
total 212
-rw-r--r--    1 go       go               0 Mar  9 14:53 go-server-perf.log
-rw-r--r--    1 go       go          105813 Mar  9 15:10 go-server.log
-rw-r--r--    1 go       go               0 Mar  9 14:53 go-shine.log
-rw-r--r--    1 go       go           26788 Mar  9 15:09 plugin-cd.go.artifact.docker.registry.log
-rw-r--r--    1 go       go            1482 Mar  9 15:09 plugin-cd.go.authentication.ldap.log
-rw-r--r--    1 go       go            1626 Mar  9 15:09 plugin-cd.go.authentication.passwordfile.log
-rw-r--r--    1 go       go           58970 Mar  9 15:09 plugin-cd.go.contrib.elasticagent.kubernetes.log

in my case there were error in plugin-cd.go.contrib.elasticagent.kubernetes.log

But also you can disable preconfigure in values.yaml

server:
  shouldPreconfigure: false

and redeploy it with

helm delete --purge gocd-app
helm install stable/gocd --name gocd-app --namespace gocd -f path/to/values.yaml

values.yaml for chart you can get there https://raw.githubusercontent.com/helm/charts/master/stable/gocd/values.yaml

-- Stas Serebrennikov

Source: StackOverflow

KQ

GoCD in Kubernetes(minikube) in Windows 10, container error CrashLoopBackOff, exit code: 111

Similar Questions

1 Answer

K
Q