K
Q

Statefulset - Possible to Skip creation of pod 0 when it fails and proceed with the next one?

3/29/2018

I currently do have a problem with the statefulset under the following condition:

  • I have a percona SQL cluster running with persistent storage and 2 nodes
  • now i do force both pods to fail.
  • first i will force pod-0 to fail
  • Afterwards i will force pod-1 to fail
  • Now the cluster is not able to recover without manual interference and possible dataloss

Why:

  • The statefulset is trying to bring pod-0 up first, however this one will not be brought online because of the following message:

    [ERROR] WSREP: It may not be safe to bootstrap the cluster from this node. It was not the last one to leave the cluster and may not contain all the updates. To force cluster bootstrap with this node, edit the grastate.dat file manually and set safe_to_bootstrap to 1

What i could do alternatively, but what i dont really like:

  • I could change ".spec.podManagementPolicy" to "Parallel" but this could lead to race conditions when forming the cluster. Thus i would like to avoid that, i basically like the idea of starting the nodes one after another

What i would like to have:

  • the possibility to have ".spec.podManagementPolicy":"OrderedReady" activated but with the possibility to adjust the order somehow
  • to be able to put specific pods into "inactive" mode so they are being ignored until i enable them again

Is something like that available? Does someone have any other ideas?

-- StokeHead
kubernetes

Similar Questions

Multiple deployment profiles with fabric8
Kubernetes set deploment number of replicas based on namespace
How do solve this error when running apache spark 2.3.0 on Kubernetes with a jar from a remote source
AKS MySQL SSL problems (nodejs,knex)

1 Answer

3/30/2018

Unfortunately, nothing like that is available in standard functions of Kubernetes.

I see only 2 options here:

  1. Use InitContainers to somehow check the current state on relaunch. That will allow you to run any code before the primary container is started so you can try to use a custom script in order to resolve the problem etc.

  2. Modify the database startup script to allow it to wait for some Environment Variable or any flag file and use PostStart hook to check the state before running a database.

But in both options, you have to write your own logic of startup order.

-- Anton Kostenko
Source: StackOverflow