Last Pod In Kubernetes Stefulsets is ready after 3 times restarting

8/1/2018

I am deploying cassandra in Kubernetes using Helm. While starting the statefulsets, For eg. with 6 pods, the last pod is starting after 3 times restart(crashloopbackoff: Backoff restarting failed container). But after 3 times restart, the pod is ready. Before that, I used PodManagementPolicy: OrderedReady. I didnt face this problem at that time. I want to start all the pods at the same time. So I give PodManagementPolicy: Parallel. Now I face this problem.

-- Arun
cassandra
kubectl
kubernetes
kubernetes-helm

1 Answer

8/1/2018

You can't start multiple Cassandra instances in parallell. Each Cassandra node has to bootstrap (stream data) and join the cluster. If a joining node notices another node is already joining it will crash (stop Cassandra). This is why you're getting the crashbackoff message.

I recommend you reverse your PodManagementPolicy and setup a readinessProbe. Example script: https://github.com/instaclustr/cassandra-operator/blob/ace024626c9339650a5a76861f36af48423a35be/docker/cassandra/readiness-probe.sh

-- Simon Fontana Oscarsson
Source: StackOverflow