I am trying to run the helm chart for mongodb and I am running into readiness / liveness probe issue. The Mongodb helm chart from the stable repository installs the pods and the services. But the pods crash because of the readiness probe. I can try removing them, but I want to know why this is happening.
helm install --name mongodb stable/mongodb
kubectl describe pod mongodb-5ff464b5b9-nnz5x
Containers:
mongodb:
Container ID: docker://055f12c14992dc111ee8f0baff7faceafae6a16e518b27a174a7997408235afa
Image: docker.io/bitnami/mongodb:4.0.2-debian-9
Image ID: docker-pullable://bitnami/mongodb@sha256:6c050bf9063976e12aabb7bafcc7b2616df1e1dbbf19ac41b4799826abec92a3
Port: 27017/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Thu, 27 Sep 2018 17:54:01 -0400
Finished: Thu, 27 Sep 2018 17:55:50 -0400
Ready: False
Restart Count: 6
Liveness: exec [mongo --eval db.adminCommand('ping')] delay=30s timeout=5s period=10s #success=1 #failure=6
Readiness: exec [mongo --eval db.adminCommand('ping')] delay=5s timeout=5s period=10s #success=1 #failure=6
Environment:
MONGODB_ROOT_PASSWORD: <set to the key 'mongodb-root-password' in secret 'mongodb'> Optional: false
MONGODB_USERNAME:
MONGODB_DATABASE:
MONGODB_ENABLE_IPV6: yes
MONGODB_EXTRA_FLAGS:
Mounts:
/bitnami/mongodb from data (rw)
In the errors section, I see the following. Do I need to do anything to open the container ports?
connecting to: mongodb://127.0.0.1:27017
2018-09-27T21:43:04.664+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 6m kubelet, vm-b768f1df-3df6-4a79-50e5-9af633c6d9b9 Readiness probe failed: MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
2018-09-27T21:43:14.670+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 6m kubelet, vm-b768f1df-3df6-4a79-50e5-9af633c6d9b9 Readiness probe failed: MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
2018-09-27T21:43:24.657+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed
Logs from the pod I checked the logs, nothing noteworthy
`kubectl logs mongodb-5ff464b5b9-nnz5x -p`
------
Welcome to the Bitnami mongodb container
Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
nami INFO Initializing mongodb
mongodb INFO ==> Deploying MongoDB from scratch...
mongodb INFO ==> No injected configuration files found. Creating default config files...Ran the same helm chart on GKE and I ran into the exact same issue
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 1m (x4 over 1m) default-scheduler PersistentVolumeClaim is not bound: "mongodb-bitnami" (repeated 3 times)
Normal Scheduled 1m default-scheduler Successfully assigned mongodb-bitnami-7b65895fd7-xr6gg to gke-cluster-default-pool-bec82955-xmhz
Normal SuccessfulMountVolume 1m kubelet, gke-cluster-default-pool-bec82955-xmhz MountVolume.SetUp succeeded for volume "default-token-whl5w"
Normal SuccessfulMountVolume 1m kubelet, gke-cluster-default-pool-bec82955-xmhz MountVolume.SetUp succeeded for volume "pvc-ddda94dd-c2c1-11e8-b2f0-42010af00119"
Normal Pulling 1m kubelet, gke-cluster-default-pool-bec82955-xmhz pulling image "docker.io/bitnami/mongodb:4.0.2-debian-9"
Normal Pulled 39s kubelet, gke-cluster-default-pool-bec82955-xmhz Successfully pulled image "docker.io/bitnami/mongodb:4.0.2-debian-9"
Normal Created 38s kubelet, gke-cluster-default-pool-bec82955-xmhz Created container
Normal Started 37s kubelet, gke-cluster-default-pool-bec82955-xmhz Started container
Warning Unhealthy 28s kubelet, gke-cluster-default-pool-bec82955-xmhz Readiness probe failed: MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
2018-09-28T01:58:44.936+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed
Warning Unhealthy 20s kubelet, gke-cluster-default-pool-bec82955-xmhz Readiness probe failed: MongoDB shell version v4.0.2
connecting to: mongodb://127.0.0.1:27017
2018-09-28T01:58:52.114+0000 E QUERY [js] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed: SocketException: Error connecting to 127.0.0.1:27017 :: caused by :: Connection refused :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed
I think this could be an issue with the bitnami helm repo. I remove that from the list of my repos, and used the mongodb from the k8s repo, and it worked fine.
It's not related to whether ports are open or not and looks more like mongo itself is crashing and the readiness probes are failing.
You can find more insights from looking at the logs:
kubectl logs <pod-crashing-name> (-c <container-name-optionally)You can also ssh into the nodes where your pods are running an get the docker logs
docker ps -a | grep mongo
docker logs <container-id-that-crashed>