We are deploying an akka cluster into kubernetes with akka-dns discovery (headless service) for cluster bootstrap.

If the first deployment fails and the pod cannot start (for any reason), and we rollout a fix, the old replicaset keeps the failing pod "alive", which is kept picked up by the headless service (publishNotReadyAddresses are necessary for the bootstrap), hence the new good pod finds it as a potential contact point for the cluster. Unfortunately, this bad pod has the lowest address among the contact points. Per the bootstrap process:

If no cluster exists, each node returns an empty list of seed-nodes. In that case, the node with the lowest address from the set of contact points forms a new cluster and starts advertising itself as a seed node.

It means that since the bad pod can't form a cluster (by joining to itself), and the new good pod don't want to form it since it does not have the lowest address, no cluster will be formed. Ever.

This can be fixed by hand by deleting the Old ReplicaSet from the Deployment, which in turn removes the bad pod, and eventually, the good pod will have the lowest - and the only - address, and the cluster is created.

(We only have a staging env yet with a single pod per service, I will test it with a replication-factor/required-contact-points of 3, but I suspect the same problem can arise.)

Is there a configuration or strategy in kubernetes or akka to prevent this behaviour so that it is not be fixed by hand or with a custom script?