How to check readiness without creating a deadlock in a cluster configuration in Kubernetes?

10/11/2021

I want to deploy a service with three replicas, each having a readiness probe. One pod will start working (sending data at a port on which readiness is tested) only when the other two pods also spin up. All the three pods need to spin up, register their IP in an internal service, and then they will be discoverable.

The readiness probe, in my opinion, works sequentially, and so, only spins up one pod. This creates a deadlock situation where the starting pod waits for the other two pods to atlease start running even if they don't start the application, and K8s does not spin up the other two pods until the readiness of the first pod is satifsfied.

My readiness config is:

readinessProbe=ExecProbe(
          execute=ExecAction(command=["curl", "localhost:2004"]),
          initialDelaySeconds=120,
          timeoutSeconds=10,
          periodSeconds=10,
          successThreshold=1,
          failureThreshold=10
        )

I want my pods to spin up, even if the current ones are running but their readiness is not successful. Or maybe I should use something other than readiness?

-- Aviral Srivastava
kubernetes
readinessprobe

1 Answer

10/11/2021

If you are using StatefulSet, use parallel pod management to allow Kubernetes to create the replicas in parallel without waiting for previous pods to be ready.

Set .spec.podManagementPolicy: Parallel in the manifest of the StatefulSet .

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql-db
spec:
  podManagementPolicy: Parallel
  replicas: 3
  <omitted>
-- Lukman
Source: StackOverflow