I'm looking to perform an automatic rolling update of a Kubernetes cluster in a way that's ignorant of the details of the applications running on the cluster. In principle, PodDisruptionBudget should facilitate this.

Here's the snag: there's an Elasticsearch cluster running on this Kubernetes cluster, and I can't find a way to correctly express the "OK to evict an ES Pod" signal. Specifically, this seems to be a case where the "this Pod can receive traffic" and "this Pod can be evicted" signals can't both be represented by readinessProbe.

This ES cluster's indices have number_of_replicas: 1, and there's a PDB with maxUnavailable: 1. The ES Pods each specify a readiness probe that requests /_cluster/health?wait_for_status=yellow.

As-is, if we evict an ES Pod, the replacement Pod will join the ES cluster, start up, and return to ready status while the ES cluster as a whole is still yellow and replicating shards (and thus it's still unsafe to evict any additional ES Pods).

Has anyone worked around this successfully? Am I misunderstanding the semantics of probes/PDBs?

Some options we've considered:

• Using wait_for_status=green in the readiness probe would mean that all ES Pods become unready when the ES cluster health is yellow.
• Increasing the ES indices' number_of_replicas to 2 only slightly reduces the probability of a rolling update damaging the ES cluster (assume these shards are slow to replicate).
• Ditto with setting a large initialDelaySeconds on the readinessProbe. It's possible for that to undershoot the time for shard replication to complete.
• Ditto with using a preStop hook (this is the approach the community Helm chart appears to take) and a long grace period.
• Decreasing the PDB's maxUnavailable to 0 means that the rolling update has to be run by a human who can remove the PDB, evaluate the status of the ES cluster, etc.
• A hypothetical, um, evictablenessProbe that checked wait_for_status=green would work, but no such API exists.