How to tell Kubernetes to not reschedule a pod unless it dies?

7/30/2021

Kubernetes tends to assume apps are small/lightweight/stateless microservices which can be stopped on one node and restarted on another node with no downtime.

We have a slow starting (20min) legacy (stateful) application which, once run as a set of pod should not be rescheduled without due cause. The reason being all user sessions will be killed and the users will have to login again. There is NO way to serialize the sessions and externalize them. We want 3 instances of the pod.

Can we tell k8s not to move a pod unless absolutely necessary (i.e. it dies)?

Additional information:

  • The app is a tomcat/java monolith
  • Assume for the sake of argument we would like to run it in Kubernetes
  • We do have a liveness test endpoint available
-- Marc
kubernetes

1 Answer

8/2/2021

There is no benefit, if you tell k8s to use only one pod. That is not the "spirit" of k8s. In this case, it might be better to use a dedicated machine for your app.
But you can assign a pod to a special node - Assigning Pods to Nodes. The should be necessary only, when special hardware requirements are needed (e.g. the AI-microservice needs a GPU, which is only on node xy).

k8s don't restart your pod for fun. It will restart it, when there is a reason (node died, app died, ...) and I never noticed a "random reschedule" in a cluster. It is hard to say, without any further information (like deployment, logs, cluster) what exactly happened to you.

And for your comment: There are different types of recreation, one of them starts a fresh instance and will kill the old one, when the startup was successfully. Look here: Kubernetes deployment strategies


All points together: 1. Don't enforce a node to your app - k8s will "smart" select the node. 2. There are normally no planned reschedules in k8s. 3. k8s will recreate pods only, if there is a reason. Maybe your app didn't answer on the liveness-endpoint? Or someone/something deleting your pod?

-- akop
Source: StackOverflow