I have been reading about liveness and readiness probes in kubernetes and I would like to use them to check and see if a cluster has come alive.
The question is how to configure a readiness probe for an entire statefulset, and not an individual pod/container.
A simple HTTP check can be used to determine readiness, but the issue I'm running into is that the readinessCheck seems to apply to the container/pod and not to the set itself.
For the software I'm using, the HTTP endpoint doesn't come up until the cluster forms; meaning that each individual pod would fail the readinessCheck until all three are up and find one another.
The behavior I'm seeing in Kubernetes right now is that the first of 3 replicas is created, and Kubernetes does not even attempt to create replicas 2 and 3 until the first passes the readinessCheck, which never happens, because all three have to be up for it to have a chance to pass it.
You need to change .spec.podManagementPolicy for a StatefulSet from OrderedReady to Parallel policy.
This way K8S will start all your pods in parallel and won't wait for probes.
From documentation
podManagementPolicy controls how pods are created during initial scale up, when replacing pods on nodes, or when scaling down. The default policy is OrderedReady, where pods are created in increasing order (pod-0, then pod-1, etc) and the controller will wait until each pod is ready before continuing. When scaling down, the pods are removed in the opposite order. The alternative policy is Parallel which will create pods in parallel to match the desired scale without waiting, and on scale down will delete all pods at once.